技术领域Technical Field
本发明涉及软件技术领域，更具体地说，涉及一种多通道数据处理方法、装置、电子设备及存储介质。The present invention relates to the field of software technology, and more specifically, to a multi-channel data processing method, device, electronic equipment and storage medium.
背景技术Background technique
现阶段，利用深度学习技术对诸如图像等多通道数据进行分类、预测等任务时，需要对其提取时间、空间等维度的特征，常使用标准卷积提取时间特征。At present, when using deep learning technology to perform classification and prediction tasks on multi-channel data such as images, it is necessary to extract features of time, space and other dimensions. Standard convolution is often used to extract time features.
而在标准卷积提取过程中，会将不同通道的数据进行交互，因此常导致提取到的时间特征存在多余噪声，这就极大降低了任务的执行效率。In the standard convolution extraction process, data from different channels will interact, which often leads to redundant noise in the extracted temporal features, greatly reducing the execution efficiency of the task.
发明内容Summary of the invention
有鉴于此，为解决上述问题，本发明提供一种多通道数据处理方法、装置、电子设备及存储介质，技术方案如下：In view of this, in order to solve the above problems, the present invention provides a multi-channel data processing method, device, electronic device and storage medium, and the technical solution is as follows:
一种多通道数据处理方法，所述多通道数据处理方法包括：A multi-channel data processing method, the multi-channel data processing method comprising:
获取待处理的多通道数据；Obtain multi-channel data to be processed;
以深度卷积的方式对所述多通道数据进行卷积运算，以获得所述多通道数据的第一特征图；Performing a convolution operation on the multi-channel data in a depthwise convolution manner to obtain a first feature map of the multi-channel data;
将所述多通道数据作为第二特征图，通过堆叠所述第一特征图与所述第二特征图进行残差连接，以提取获得所述多通道数据的时间特征。The multi-channel data is used as a second feature map, and the first feature map is stacked and residually connected with the second feature map to extract the time features of the multi-channel data.
优选的，所述获取待处理的多通道数据，包括：Preferably, the obtaining of the multi-channel data to be processed includes:
获取多通道采样的脑电数据。Acquire multi-channel sampled EEG data.
优选的，所述以深度卷积的方式对所述多通道数据进行卷积运算，包括：Preferably, performing a convolution operation on the multi-channel data in a depthwise convolution manner includes:
调取多尺度深度卷积层，所述多尺度深度卷积层由多个卷积核尺寸不同的深度卷积层组成；Retrieving a multi-scale deep convolutional layer, wherein the multi-scale deep convolutional layer is composed of a plurality of deep convolutional layers with different convolution kernel sizes;
针对所述多尺度深度卷积层中的每个深度卷积层来说，以该深度卷积层对应的卷积核尺寸提取所述脑电数据在各通道下的特征，得到子特征图；For each deep convolution layer in the multi-scale deep convolution layer, extract the features of the EEG data in each channel with the convolution kernel size corresponding to the deep convolution layer to obtain a sub-feature map;
将所述多尺度深度卷积层中各深度卷积层的子特征图进行堆叠。The sub-feature maps of each depth convolution layer in the multi-scale depth convolution layer are stacked.
优选的，所述多尺度深度卷积层的确定方式，包括：Preferably, the method for determining the multi-scale deep convolutional layer includes:
获取所述脑电数据的多个滤波频率段和采样频率，所述多个滤波频率段中的一个滤波频率段对应一个深度卷积层；Acquire multiple filter frequency segments and sampling frequencies of the EEG data, wherein one filter frequency segment among the multiple filter frequency segments corresponds to a deep convolution layer;
针对所述多个滤波频率段中的每个滤波频率段来说，根据该滤波频率段中的最低频率与所述采样频率，计算该滤波频率段所对应深度卷积层的卷积核尺寸。For each of the plurality of filter frequency segments, a convolution kernel size of a depthwise convolution layer corresponding to the filter frequency segment is calculated according to a lowest frequency in the filter frequency segment and the sampling frequency.
优选的，所述多通道数据处理方法还包括：以空域卷积的方式对提取时间特征后的所述脑电数据进行卷积运算，以提取获得所述脑电数据的空间特征。Preferably, the multi-channel data processing method further comprises: performing a convolution operation on the EEG data after extracting the time features in a spatial domain convolution manner to extract the spatial features of the EEG data.
一种多通道数据处理装置，所述多通道数据处理装置包括：A multi-channel data processing device, the multi-channel data processing device comprising:
数据获取模块，用于获取待处理的多通道数据；A data acquisition module, used for acquiring multi-channel data to be processed;
卷积运算模块，用于以深度卷积的方式对所述多通道数据进行卷积运算，以获得所述多通道数据的第一特征图；A convolution operation module, configured to perform a convolution operation on the multi-channel data in a depthwise convolution manner to obtain a first feature map of the multi-channel data;
残差连接模块，用于将所述多通道数据作为第二特征图，通过堆叠所述第一特征图与所述第二特征图进行残差连接，以提取获得所述多通道数据的时间特征。A residual connection module is used to use the multi-channel data as a second feature map, and perform a residual connection by stacking the first feature map and the second feature map to extract the time features of the multi-channel data.
优选的，所述数据获取模块，具体用于：Preferably, the data acquisition module is specifically used for:
获取多通道采样的脑电数据。Acquire multi-channel sampled EEG data.
优选的，所述卷积运算模块，具体用于：Preferably, the convolution operation module is specifically used for:
调取多尺度深度卷积层，所述多尺度深度卷积层由多个卷积核尺寸不同的深度卷积层组成；针对所述多尺度深度卷积层中的每个深度卷积层来说，以该深度卷积层对应的卷积核尺寸提取所述脑电数据在各通道下的特征，得到子特征图；将所述多尺度深度卷积层中各深度卷积层的子特征图进行堆叠。A multi-scale deep convolution layer is retrieved, wherein the multi-scale deep convolution layer is composed of a plurality of deep convolution layers with different convolution kernel sizes; for each deep convolution layer in the multi-scale deep convolution layer, the features of the EEG data under each channel are extracted with the convolution kernel size corresponding to the deep convolution layer to obtain a sub-feature map; and the sub-feature maps of each deep convolution layer in the multi-scale deep convolution layer are stacked.
一种电子设备，所述电子设备包括：至少一个存储器和至少一个处理器；所述存储器存储有应用程序，所述处理器调用所述存储器存储的应用程序，所述应用程序用于实现本发明实施例提供的任一种多通道数据处理方法。An electronic device comprises: at least one memory and at least one processor; the memory stores an application program, the processor calls the application program stored in the memory, and the application program is used to implement any multi-channel data processing method provided in an embodiment of the present invention.
一种存储介质，所述存储介质存储有计算机程序代码，所述计算机程序代码执行时实现本发明实施例提供的任一种多通道数据处理方法。A storage medium stores computer program code, and when the computer program code is executed, any multi-channel data processing method provided by an embodiment of the present invention is implemented.
相较于现有技术，本发明实现的有益效果为：Compared with the prior art, the present invention has the following beneficial effects:
本发明提供一种多通道数据处理方法、装置、电子设备及存储介质，首先获取待处理的多通道数据；进而以深度卷积的方式对多通道数据进行卷积运算，以获得多通道数据的第一特征图；进一步将所述多通道数据作为第二特征图，再通过堆叠第一特征图与第二特征图进行残差连接，以提取获得多通道数据的时间特征。也就是说，为避免不同通道之间数据交互导致的多余噪声，本发明采用深度卷积的方式对多通道数据进行卷积运算，以此保证通道间的独立性，另外，为保持多通道数据原本的时空结构，本发明采用残差连接，将源数据作为一个独立的特征图与深度卷积得到的特征图进行堆叠，以此保证原有空间特征不被破坏，最终能够更准确地提取到多通道数据中所蕴含的固有时间特征，保证整个任务的准确性。The present invention provides a multi-channel data processing method, device, electronic device and storage medium, firstly obtaining the multi-channel data to be processed; then performing convolution operation on the multi-channel data in the manner of deep convolution to obtain a first feature map of the multi-channel data; further using the multi-channel data as a second feature map, and then performing residual connection by stacking the first feature map and the second feature map to extract the time characteristics of the multi-channel data. In other words, in order to avoid the redundant noise caused by the interaction of data between different channels, the present invention performs convolution operation on the multi-channel data in the manner of deep convolution to ensure the independence between channels. In addition, in order to maintain the original spatiotemporal structure of the multi-channel data, the present invention uses residual connection to stack the source data as an independent feature map with the feature map obtained by deep convolution, so as to ensure that the original spatial characteristics are not destroyed, and finally the inherent time characteristics contained in the multi-channel data can be more accurately extracted to ensure the accuracy of the entire task.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
图1为本发明实施例提供的多通道数据处理方法的方法流程图；FIG1 is a flow chart of a multi-channel data processing method provided by an embodiment of the present invention;
图2为本发明实施例提供的多通道数据处理方法的部分方法流程图；FIG2 is a partial method flow chart of a multi-channel data processing method provided by an embodiment of the present invention;
图3为本发明实施例提供的多尺度深度卷积过程的流程示意图；FIG3 is a schematic diagram of a flow chart of a multi-scale deep convolution process provided by an embodiment of the present invention;
图4为本发明实施例提供的SEED数据集上各对照方法与本方法的准确率折线图；FIG4 is a line graph showing the accuracy of various control methods and the present method on the SEED dataset provided by an embodiment of the present invention;
图5为本发明实施例提供的SEED-V数据集中上各对照方法与本方法的准确率折线图；FIG5 is a line graph showing the accuracy of various control methods and the present method in the SEED-V dataset provided by an embodiment of the present invention;
图6为本发明实施例提供的多通道数据处理装置的结构示意图。FIG6 is a schematic diagram of the structure of a multi-channel data processing device provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本发明作进一步详细的说明。In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
参见图1，图1为本发明实施例提供的多通道数据处理方法的方法流程图。如图1所示，该多通道数据处理方法包括如下步骤：Referring to Fig. 1, Fig. 1 is a flowchart of a multi-channel data processing method provided by an embodiment of the present invention. As shown in Fig. 1, the multi-channel data processing method comprises the following steps:
S10，获取待处理的多通道数据。S10, obtaining multi-channel data to be processed.
本发明实施例中，多通道数据即包含多个通道的数据，举例来说，在图像识别领域，RGB图像即为由R通道、G通道和B通道所组成的多通道数据。继续举例来说，在脑电情绪识别领域，脑电数据可以通过多个电极片进行采样，而一个电极片可以对应为一个通道，因此脑电数据也可以为多通道数据。In the embodiment of the present invention, multi-channel data is data containing multiple channels. For example, in the field of image recognition, an RGB image is multi-channel data consisting of an R channel, a G channel, and a B channel. For another example, in the field of EEG emotion recognition, EEG data can be sampled through multiple electrodes, and one electrode can correspond to one channel, so the EEG data can also be multi-channel data.
对此，步骤S10“获取待处理的多通道数据”可以采用如下步骤：In this regard, step S10 "obtaining multi-channel data to be processed" may adopt the following steps:
获取多通道采样的脑电数据。Acquire multi-channel sampled EEG data.
本发明实施例提供的多通道数据处理方法应用于脑电情绪识别领域时，可以更准确地提取脑电数据中蕴含的多维度情绪特征的固有时间表征，从而更全面地捕捉脑电数据中涵盖的多种维度的情感信息，实现情绪的精确辨别。When the multi-channel data processing method provided by the embodiment of the present invention is applied to the field of EEG emotion recognition, it can more accurately extract the inherent time representation of the multi-dimensional emotional features contained in the EEG data, thereby more comprehensively capturing the emotional information of multiple dimensions covered in the EEG data and achieving accurate identification of emotions.
S20，以深度卷积的方式对多通道数据进行卷积运算，以获得多通道数据的第一特征图。S20, performing a convolution operation on the multi-channel data in a depthwise convolution manner to obtain a first feature map of the multi-channel data.
为方便理解本发明实施例，以脑电数据作为多通道数据进行说明。本发明实施例中，可以通过深度卷积层对脑电数据进行卷积运算，在卷积运算过程中，深度卷积层分别对脑电数据在各通道的数据进行卷积运算，再将所有通道的卷积运算结果组合为一个特征图，即第一特征图，这就保证了通道间的独立性。To facilitate understanding of the embodiments of the present invention, EEG data is used as multi-channel data for explanation. In the embodiments of the present invention, a deep convolutional layer can be used to perform convolution operations on the EEG data. During the convolution operation process, the deep convolutional layer performs convolution operations on the EEG data in each channel respectively, and then combines the convolution operation results of all channels into a feature map, i.e., the first feature map, which ensures the independence between channels.
相比单一深度卷积层，多尺度深度卷积层可以得到更丰富的多尺度时间特征。对此，具体实现过程中，可以采用多尺度深度卷积技术对多通道数据进行卷积运算。对此，步骤S20“以深度卷积的方式对多通道数据进行卷积运算”可以采用如下步骤，方法流程图如图2所示：Compared with a single deep convolution layer, a multi-scale deep convolution layer can obtain richer multi-scale temporal features. In this regard, in the specific implementation process, a multi-scale deep convolution technique can be used to perform convolution operations on multi-channel data. In this regard, step S20 "performing convolution operations on multi-channel data in a deep convolution manner" can adopt the following steps, and the method flow chart is shown in Figure 2:
S201，调取多尺度深度卷积层，多尺度深度卷积层由多个卷积核尺寸不同的深度卷积层组成。S201, call a multi-scale deep convolutional layer, the multi-scale deep convolutional layer is composed of multiple deep convolutional layers with different convolution kernel sizes.
S202，针对多尺度深度卷积层中的每个深度卷积层来说，以该深度卷积层对应的卷积核尺寸提取脑电数据在各通道下的特征，得到子特征图。S202, for each deep convolution layer in the multi-scale deep convolution layer, extract the features of the EEG data in each channel using the convolution kernel size corresponding to the deep convolution layer to obtain a sub-feature map.
S203，将多尺度深度卷积层中各深度卷积层的子特征图进行堆叠。S203, stacking the sub-feature maps of each deep convolutional layer in the multi-scale deep convolutional layer.
为方便理解本发明实施例，继续以脑电数据作为多通道数据进行说明。在脑电情绪识别领域，有很多研究是利用多尺度卷积对脑电数据进行时间特征的提取，例如TSception（一种基于脑电图的时间动态和空间不对称性情感识别方法），利用多个卷积核尺寸不同的标准卷积层进行提取，将多个卷积核尺寸下的时间特征在指定的特征维度上进行连接，这会将不同通道的数据进行交互，因此可能会导致提取到的时间特征存在多余噪声。同时，为保证源数据中时间特征不被破坏，例如运动想象方向的EEGSym（一种用深度学习克服基于运动意象的脑机接口的主体间变异性的方法）采用残差连接，具体做法是将源数据与多个卷积核尺寸下的时间特征对应相加，虽然其论文中的实验也验证了这种残差连接方式的有效性，但残差连接方式仍存在很大的可探索空间。对此，本发明实施例提供一种特征图叠加的方式进行残差连接，通过多尺度深度卷积对脑电数据卷积运算得到特征图，再将脑电数据的源数据作为一个独立特征图进行堆叠，采用特征图堆叠的方式可以保证每个特征图独特的时空结构不变，同时也未破坏脑电数据中的时空结构，从而此保证原有时空特征不被破坏。To facilitate understanding of the embodiments of the present invention, EEG data is continued to be used as multi-channel data for explanation. In the field of EEG emotion recognition, there are many studies that use multi-scale convolution to extract temporal features from EEG data, such as TSception (a method for emotion recognition based on temporal dynamics and spatial asymmetry of EEG), which uses multiple standard convolution layers with different convolution kernel sizes for extraction, and connects the temporal features under multiple convolution kernel sizes on the specified feature dimension, which will interact with the data of different channels, and thus may cause the extracted temporal features to have excess noise. At the same time, in order to ensure that the temporal features in the source data are not destroyed, for example, EEGSym (a method for overcoming the inter-subject variability of brain-computer interfaces based on motor imagery using deep learning) in the direction of motor imagery uses residual connection, which specifically adds the source data to the temporal features under multiple convolution kernel sizes. Although the experiments in the paper also verify the effectiveness of this residual connection method, there is still a lot of room for exploration in the residual connection method. In this regard, an embodiment of the present invention provides a feature map stacking method to perform residual connection, and obtains a feature map by convolution operation on the EEG data through multi-scale deep convolution, and then stacks the source data of the EEG data as an independent feature map. The feature map stacking method can ensure that the unique spatiotemporal structure of each feature map remains unchanged, and at the same time, the spatiotemporal structure in the EEG data is not destroyed, thereby ensuring that the original spatiotemporal features are not destroyed.
参见图3，图3为本发明实施例提供的多尺度深度卷积过程的流程示意图。如图3所示，多尺度深度卷积层由多个深度卷积层组成，即深度卷积层1~n，在本发明实施例中每个深度卷积层的卷积核尺寸不同，这就可以提取不同尺度的时间特征，而采用深度卷积层对脑电数据进行卷积运算，又可以保证不同通道之间的数据不产生交互。See Figure 3, which is a flow chart of the multi-scale deep convolution process provided by an embodiment of the present invention. As shown in Figure 3, the multi-scale deep convolution layer is composed of multiple deep convolution layers, namely deep convolution layers 1 to n. In the embodiment of the present invention, the convolution kernel size of each deep convolution layer is different, which can extract time features of different scales. The deep convolution layer is used to perform convolution operations on EEG data, which can ensure that data between different channels do not interact.
由此，对于多尺度深度卷积层中的每个深度卷积层来说，以该深度卷积层对应的卷积核尺寸对脑电数据各通道下的特征进行卷积运算，以此得到由各通道的卷积运算结果所组成的特征图，即子特征图。需要说明的是，深度卷积层进行卷积运算的过程为已知方案，在此不再赘述。Therefore, for each deep convolution layer in the multi-scale deep convolution layer, the convolution kernel size corresponding to the deep convolution layer is used to perform convolution operation on the features of each channel of the EEG data, so as to obtain a feature map composed of the convolution operation results of each channel, that is, a sub-feature map. It should be noted that the process of performing convolution operation on the deep convolution layer is a known scheme and will not be repeated here.
以多尺度深度卷积层包括四个深度卷积层、、、为例来说明，可以采用如下公式（1）来表示深度卷积的过程：The multi-scale deep convolutional layer includes four deep convolutional layers , , , For example, the following formula (1) can be used to represent the process of deep convolution:
（1） (1)
其中，表示卷积运算结果，表示深度卷积层对应的卷积核，表示卷积运算的输入，表示对深度卷积层设置的卷积核数量。in, represents the result of the convolution operation, Represents the convolution kernel corresponding to the depth convolution layer, represents the input of the convolution operation, Indicates the number of convolution kernels set for the depthwise convolution layer.
进一步，在获得多尺度深度卷积层中各深度卷积层对应的子特征图之后，可以将所有深度卷积层的子特征图进行堆叠，以此实现多尺度深度卷积层的特征连接，以此得到多通道数据的第一特征图。Furthermore, after obtaining the sub-feature maps corresponding to each deep convolutional layer in the multi-scale deep convolutional layer, the sub-feature maps of all the deep convolutional layers can be stacked to achieve feature connection of the multi-scale deep convolutional layer, so as to obtain the first feature map of the multi-channel data.
需要说明的是，多个尺度深度卷积层中各深度卷积层的卷积核数量是相同的，卷积核数量可以根据实际场景进行设置，本发明实施例对此不做限定。It should be noted that the number of convolution kernels in each depth convolution layer in multiple scale depth convolution layers is the same, and the number of convolution kernels can be set according to the actual scenario, which is not limited in the embodiment of the present invention.
在实际应用中，深度卷积层的卷积核尺寸是多尺度深度卷积实现的关键特征，对此，本发明实施例中可以根据脑电数据的滤波频率段和采样频率来确定，以此保证多尺度深度卷积层能够适配不同的脑电情绪识别场景。对此，本发明实施例中，多尺度深度卷积层的确定方式，包括：In practical applications, the convolution kernel size of the deep convolution layer is a key feature of the multi-scale deep convolution. In this regard, the embodiment of the present invention can be determined based on the filter frequency segment and sampling frequency of the EEG data to ensure that the multi-scale deep convolution layer can adapt to different EEG emotion recognition scenarios. In this regard, in the embodiment of the present invention, the method for determining the multi-scale deep convolution layer includes:
获取脑电数据的多个滤波频率段和采样频率，多个滤波频率段中的一个滤波频率段对应一个深度卷积层；针对多个滤波频率段中的每个滤波频率段来说，根据该滤波频率段中的最低频率与采样频率，计算该滤波频率段所对应深度卷积层的卷积核尺寸。Acquire multiple filter frequency segments and sampling frequencies of EEG data, wherein one filter frequency segment in the multiple filter frequency segments corresponds to a deep convolution layer; for each filter frequency segment in the multiple filter frequency segments, calculate the convolution kernel size of the deep convolution layer corresponding to the filter frequency segment according to the lowest frequency in the filter frequency segment and the sampling frequency.
具体的，本发明实施例中，多尺度深度卷积层中深度卷积层的数量可以由脑电数据滤波的频率段（即滤波频率段）所确定，一个滤波频率段对应一个深度卷积层，即滤波频率段的数量作为多尺度深度卷积层中深度卷积层的数量。举例来说，某一脑电情绪识别场景下，脑电数据的滤波范围为4HZ ~47HZ，包含了四个滤波频率段4HZ~8HZ、8HZ~13HZ、13HZ~30HZ、30HZ ~47HZ，此时即可确定多尺度深度卷积层中深度卷积层的数量为四。Specifically, in an embodiment of the present invention, the number of deep convolutional layers in the multi-scale deep convolutional layer can be determined by the frequency band of EEG data filtering (i.e., filtering frequency band), and one filtering frequency band corresponds to one deep convolutional layer, that is, the number of filtering frequency bands is used as the number of deep convolutional layers in the multi-scale deep convolutional layer. For example, in a certain EEG emotion recognition scenario, the filtering range of EEG data is 4HZ ~ 47HZ, which includes four filtering frequency bands 4HZ ~ 8HZ, 8HZ ~ 13HZ, 13HZ ~ 30HZ, and 30HZ ~ 47HZ. At this time, it can be determined that the number of deep convolutional layers in the multi-scale deep convolutional layer is four.
另外，对于每个滤波频率段来说，可以根据该滤波频率段中的最低频率与采样频率来计算该滤波频率段所对应深度卷积层的卷积核尺寸，具体可以将采样频率与最低频率的比值作为卷积核尺寸。以4HZ~8HZ这一滤波频率段举例来说，如果采样率为200，则卷积核尺寸为200/4=50，此时卷积核尺寸可以表示为1*50，其中1表示输入通道为1。In addition, for each filter frequency segment, the convolution kernel size of the depth convolution layer corresponding to the filter frequency segment can be calculated according to the lowest frequency in the filter frequency segment and the sampling frequency. Specifically, the ratio of the sampling frequency to the lowest frequency can be used as the convolution kernel size. Taking the filter frequency segment of 4HZ~8HZ as an example, if the sampling rate is 200, the convolution kernel size is 200/4=50. At this time, the convolution kernel size can be expressed as 1*50, where 1 indicates that the input channel is 1.
S30，将多通道数据作为第二特征图，通过堆叠第一特征图与第二特征图进行残差连接，以提取获得多通道数据的时间特征。S30, using the multi-channel data as a second feature map, and performing a residual connection by stacking the first feature map and the second feature map to extract the time features of the multi-channel data.
为方便理解本发明实施例，继续以脑电数据作为多通道数据进行说明。本发明实施例中，由于脑电数据的维度与经多尺度深度卷积后的特征图的形状是一致的，因此可以将脑电数据作为一个独立特征图（即第二特征图）与第一特征图进行堆叠，以此完成残差连接，堆叠结果即残差连接结果作为脑电数据的时间特征。To facilitate understanding of the embodiments of the present invention, EEG data is used as multi-channel data for further explanation. In the embodiments of the present invention, since the dimension of the EEG data is consistent with the shape of the feature map after multi-scale deep convolution, the EEG data can be stacked with the first feature map as an independent feature map (i.e., the second feature map) to complete the residual connection, and the stacking result, i.e., the residual connection result, is used as the temporal feature of the EEG data.
另外，由于本发明实施例中脑电数据经多尺度深度卷积与残差连接后的时空特征未被破坏，因此，本发明实施例提供的多通道数据处理方法还可以包括如下步骤：In addition, since the spatiotemporal features of the EEG data after multi-scale deep convolution and residual connection are not destroyed in the embodiment of the present invention, the multi-channel data processing method provided by the embodiment of the present invention may also include the following steps:
以空域卷积的方式对提取时间特征后的脑电数据进行卷积运算，以提取获得脑电数据的空间特征。The EEG data after temporal features are extracted is convolved in the form of spatial convolution to extract the spatial features of the EEG data.
为方便理解本发明实施例，继续以脑电数据作为多通道数据进行说明。本发明实施例中，脑电数据经时间特征提取后，还可以进行空间特征提取，具体可以以空域卷积的方式对提取时间特征后的脑电数据进行卷积运算，即使用空域卷积层进行卷积运算，以此得到该脑电数据的空间特征。To facilitate understanding of the embodiments of the present invention, EEG data is used as multi-channel data for further explanation. In the embodiments of the present invention, after the temporal features of the EEG data are extracted, spatial features can also be extracted. Specifically, the EEG data after the temporal features are extracted can be convolved in a spatial domain convolution manner, that is, a spatial domain convolution layer is used to perform convolution operations, so as to obtain the spatial features of the EEG data.
需要说明的是，空域卷积层进行卷积运算的过程为已知方案，在此不再赘述，并且，空域卷积层的设置可以根据实际场景来选择，本发明实施例对此不做限定。It should be noted that the process of performing convolution operation in the spatial convolution layer is a known solution and will not be described in detail here. In addition, the setting of the spatial convolution layer can be selected according to the actual scenario, and the embodiment of the present invention is not limited to this.
还需要说明的是，上述提供一种脑电数据先提取时间特征、再提取空间特征的方案，而在实际应用中，根据不同的脑电情绪识别场景，还可以在时间特征提取前设置特征提取层，还可以将空间特征的提取设置在时间特征提取前，而无论是哪种实现方案，均不影响本发明实施例中对时间特征提取的效果。It should also be noted that the above provides a solution for extracting time features first and then spatial features of EEG data. In actual applications, according to different EEG emotion recognition scenarios, a feature extraction layer can be set before time feature extraction, and the extraction of spatial features can also be set before time feature extraction. No matter which implementation scheme is used, it does not affect the effect of time feature extraction in the embodiments of the present invention.
此外，为验证本发明的效果，发明人将本发明实施例提供的多通道数据处理方法应用于脑电情绪识别领域，在两个公开的情绪数据集SEED（The SJTU Emotion EEGDataset）和SEED-V（The SJTU Emotion EEG Dataset-V）上进行了广泛试验来验证。采用的深度学习模型为En-ESTRNet、对照方法分别为deepConvNet、EEGNet、TSception、CLISA、DE+MLP、En-ESTRNet-NTDI（其中En-ESTRNet-NTDI表示不采用多尺度深度卷积的对照模型）。In addition, to verify the effect of the present invention, the inventor applied the multi-channel data processing method provided by the embodiment of the present invention to the field of EEG emotion recognition, and conducted extensive experiments on two public emotion datasets SEED (The SJTU Emotion EEGDataset) and SEED-V (The SJTU Emotion EEG Dataset-V) to verify. The deep learning model used is En-ESTRNet, and the control methods are deepConvNet, EEGNet, TSception, CLISA, DE+MLP, and En-ESTRNet-NTDI (where En-ESTRNet-NTDI represents a control model that does not use multi-scale deep convolution).
参见表1和表2，表1记录在SEED数据集上各对照方法与本方法（En-ESTRNet）在评价指标（准确率和标准差）的试验对照结果；表2记录在SEED-V数据集上各对照方法与本方法（En-ESTRNet）在评价指标（准确率和标准差）的试验对照结果，其中，准确率表示在数据集中所有被测试数据的平均准确率、标准差表示在数据集中所有被测试数据的准确率标准差。显然，在SEED数据集上本方法在准确率和标准差均高于对照方法，而在SEED-V数据集上本方案的准确率也高于其他对照方法。See Table 1 and Table 2. Table 1 records the experimental comparison results of the evaluation indicators (accuracy and standard deviation) of each control method and this method (En-ESTRNet) on the SEED dataset; Table 2 records the experimental comparison results of the evaluation indicators (accuracy and standard deviation) of each control method and this method (En-ESTRNet) on the SEED-V dataset, where the accuracy represents the average accuracy of all tested data in the dataset, and the standard deviation represents the standard deviation of the accuracy of all tested data in the dataset. Obviously, on the SEED dataset, this method is higher than the control method in both accuracy and standard deviation, and on the SEED-V dataset, the accuracy of this solution is also higher than other control methods.
表1Table 1
表2Table 2
参见图4和图5，图4为本发明实施例提供的SEED数据集上各对照方法与本方法的准确率折线图；图5为本发明实施例提供的SEED-V数据集中上各对照方法与本方法的准确率折线图。继续参见图4和图5，横坐标表示被测数据的标识（1、2、3、4、5、……）、纵坐标表示准确率，即图4中的折线表示SEED数据集中各被测数据在各对照方法与本方法的准确率，图5中的折线表示SEED-V数据集中各被测数据在各对照方法与本方法的准确率。从上述结果中可以看出，本发明可以有效提升深度学习模型的性能，本发明在脑电数据的时间特征提取方面具有明显优势。Referring to Figures 4 and 5, Figure 4 is a line graph of the accuracy of each control method and the present method on the SEED dataset provided by an embodiment of the present invention; Figure 5 is a line graph of the accuracy of each control method and the present method on the SEED-V dataset provided by an embodiment of the present invention. Continuing to refer to Figures 4 and 5, the horizontal axis represents the identification of the tested data (1, 2, 3, 4, 5, ...), and the vertical axis represents the accuracy, that is, the broken line in Figure 4 represents the accuracy of each tested data in the SEED dataset in each control method and the present method, and the broken line in Figure 5 represents the accuracy of each tested data in the SEED-V dataset in each control method and the present method. It can be seen from the above results that the present invention can effectively improve the performance of the deep learning model, and the present invention has obvious advantages in the extraction of temporal features of EEG data.
经由以上描述，本发明实施例提供的多通道数据处理方法，具有如下优势：Based on the above description, the multi-channel data processing method provided by the embodiment of the present invention has the following advantages:
1）可以获取到更加优质的时间特征。采用深度卷积的方式，可以保证通道与通道之间的时间特征相互独立，可以在一定程度上保证数据空间信息的完整性。1) Better time features can be obtained. The deep convolution method can ensure that the time features between channels are independent of each other, and can ensure the integrity of data space information to a certain extent.
2）保证了数据时空结构的完整性。通过残差连接将源数据的特征图与多尺度深度卷积的特征图进行堆叠，保证了源数据中原始时空结构的完整性。2) The integrity of the spatiotemporal structure of the data is ensured. The feature map of the source data is stacked with the feature map of the multi-scale deep convolution through residual connection, which ensures the integrity of the original spatiotemporal structure in the source data.
3）对不同尺度特征的时空结构进行了探索。通过多尺度深度卷积得到的多个特征图都有其独特的时空结构，这些时空结构与源数据中的时空结构有较大区别，但是可能仍包含了对于后续分类有用的信息，采用特征图堆叠的方式，可以保持每个特征图独特的时空结构不变，同时也未破坏源数据中的时空结构。3) The spatiotemporal structure of features at different scales was explored. The multiple feature maps obtained by multi-scale deep convolution have their own unique spatiotemporal structures, which are quite different from those in the source data, but may still contain useful information for subsequent classification. The stacking of feature maps can keep the unique spatiotemporal structure of each feature map unchanged, while not destroying the spatiotemporal structure in the source data.
基于上述实施例提供的多通道数据处理方法，本发明实施例则对应提供执行上述多通道数据处理方法的装置，该多通道数据处理装置的结构示意图如图6所示，包括：Based on the multi-channel data processing method provided in the above embodiment, an embodiment of the present invention provides a device for executing the multi-channel data processing method. The structural schematic diagram of the multi-channel data processing device is shown in FIG6 , including:
数据获取模块10，用于获取待处理的多通道数据；A data acquisition module 10 is used to acquire multi-channel data to be processed;
卷积运算模块20，用于以深度卷积的方式对多通道数据进行卷积运算，以获得多通道数据的第一特征图；A convolution operation module 20, configured to perform a convolution operation on the multi-channel data in a depthwise convolution manner to obtain a first feature map of the multi-channel data;
残差连接模块30，用于将多通道数据作为第二特征图，通过堆叠第一特征图与第二特征图进行残差连接，以提取获得多通道数据的时间特征。The residual connection module 30 is used to use the multi-channel data as the second feature map, and perform a residual connection by stacking the first feature map and the second feature map to extract the time features of the multi-channel data.
可选的，数据获取模块10，具体用于：Optionally, the data acquisition module 10 is specifically used for:
获取多通道采样的脑电数据。Acquire multi-channel sampled EEG data.
可选的，卷积运算模块20，具体用于：Optionally, the convolution operation module 20 is specifically used for:
调取多尺度深度卷积层，多尺度深度卷积层由多个卷积核尺寸不同的深度卷积层组成；针对多尺度深度卷积层中的每个深度卷积层来说，以该深度卷积层对应的卷积核尺寸提取脑电数据在各通道下的特征，得到子特征图；将多尺度深度卷积层中各深度卷积层的子特征图进行堆叠。A multi-scale deep convolutional layer is retrieved, and the multi-scale deep convolutional layer is composed of multiple deep convolutional layers with different convolution kernel sizes. For each deep convolutional layer in the multi-scale deep convolutional layer, the features of the EEG data in each channel are extracted with the convolution kernel size corresponding to the deep convolutional layer to obtain a sub-feature map. The sub-feature maps of each deep convolutional layer in the multi-scale deep convolutional layer are stacked.
可选的，多尺度深度卷积层的确定方式，包括：Optional methods for determining a multi-scale deep convolutional layer include:
获取脑电数据的多个滤波频率段和采样频率，多个滤波频率段中的一个滤波频率段对应一个深度卷积层；针对多个滤波频率段中的每个滤波频率段来说，根据该滤波频率段中的最低频率与采样频率，计算该滤波频率段所对应深度卷积层的卷积核尺寸。Acquire multiple filter frequency segments and sampling frequencies of EEG data, wherein one filter frequency segment in the multiple filter frequency segments corresponds to a deep convolution layer; for each filter frequency segment in the multiple filter frequency segments, calculate the convolution kernel size of the deep convolution layer corresponding to the filter frequency segment according to the lowest frequency in the filter frequency segment and the sampling frequency.
可选的，上述多通道数据处理装置还包括：Optionally, the multi-channel data processing device further includes:
空间特征提取模块，用于以空域卷积的方式对提取时间特征后的脑电数据进行卷积运算，以提取获得脑电数据的空间特征。The spatial feature extraction module is used to perform convolution operation on the EEG data after the temporal features are extracted in a spatial domain convolution manner to extract the spatial features of the EEG data.
需要说明的是，本发明实施例中各模块的细化功能可以参见上述多通道数据处理方法实施例对应公开部分，在此不再赘述。It should be noted that the detailed functions of each module in the embodiment of the present invention can be found in the corresponding disclosed part of the above-mentioned multi-channel data processing method embodiment, and will not be repeated here.
基于上述实施例提供的多通道数据处理方法，本发明实施例还提供一种电子设备，电子设备包括：至少一个存储器和至少一个处理器；存储器存储有应用程序，处理器调用存储器存储的应用程序，应用程序用于实现多通道数据处理方法。Based on the multi-channel data processing method provided in the above embodiment, an embodiment of the present invention further provides an electronic device, the electronic device comprising: at least one memory and at least one processor; the memory stores an application, the processor calls the application stored in the memory, and the application is used to implement the multi-channel data processing method.
基于上述实施例提供的多通道数据处理方法，本发明实施例还提供一种存储介质，存储介质存储有计算机程序代码，计算机程序代码执行时实现多通道数据处理方法。Based on the multi-channel data processing method provided in the above embodiment, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores computer program code, and the multi-channel data processing method is implemented when the computer program code is executed.
以上对本发明所提供的一种多通道数据处理方法、装置、电子设备及存储介质进行了详细介绍，本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。The above is a detailed introduction to a multi-channel data processing method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
需要说明的是，本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备所固有的要素，或者是还包括为这些过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device that includes a series of elements is inherent to the elements, or also includes elements inherent to these processes, methods, articles or devices. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本发明。对这些实施例的多种修改对本领域的专业技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本发明的精神或范围的情况下，在其它实施例中实现。因此，本发明将不会被限制于本文所示的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410338823.9ACN117972395B (en) | 2024-03-22 | 2024-03-22 | Multi-channel data processing method and device, electronic equipment and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410338823.9ACN117972395B (en) | 2024-03-22 | 2024-03-22 | Multi-channel data processing method and device, electronic equipment and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN117972395Atrue CN117972395A (en) | 2024-05-03 |
| CN117972395B CN117972395B (en) | 2024-07-09 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410338823.9AActiveCN117972395B (en) | 2024-03-22 | 2024-03-22 | Multi-channel data processing method and device, electronic equipment and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN117972395B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190150774A1 (en)* | 2016-05-11 | 2019-05-23 | Mayo Foundation For Medical Education And Research | Multiscale brain electrode devices and methods for using the multiscale brain electrodes |
| CN111317468A (en)* | 2020-02-27 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Electroencephalogram signal classification method and device, computer equipment and storage medium |
| CN114358130A (en)* | 2021-12-08 | 2022-04-15 | 金华市伊凯动力科技有限公司 | Method for intelligent control of electric chair through brain-electricity-eye fusion |
| CN117574989A (en)* | 2023-11-10 | 2024-02-20 | 深圳睿瀚医疗科技有限公司 | Deep learning method of adaptive filtering in motor imagery classification |
| CN117609852A (en)* | 2023-10-13 | 2024-02-27 | 曲阜师范大学 | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190150774A1 (en)* | 2016-05-11 | 2019-05-23 | Mayo Foundation For Medical Education And Research | Multiscale brain electrode devices and methods for using the multiscale brain electrodes |
| CN111317468A (en)* | 2020-02-27 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Electroencephalogram signal classification method and device, computer equipment and storage medium |
| CN114358130A (en)* | 2021-12-08 | 2022-04-15 | 金华市伊凯动力科技有限公司 | Method for intelligent control of electric chair through brain-electricity-eye fusion |
| CN117609852A (en)* | 2023-10-13 | 2024-02-27 | 曲阜师范大学 | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals |
| CN117574989A (en)* | 2023-11-10 | 2024-02-20 | 深圳睿瀚医疗科技有限公司 | Deep learning method of adaptive filtering in motor imagery classification |
| Title |
|---|
| 乔大雷 等: "《海上态势视觉感知方法研究》", 31 October 2021, 长春：吉林大学出版社, pages: 28* |
| 秦宏帅: ""基于深度卷积神经网络的癫痫脑电信号识别算法研究"", 《中国优秀硕士学位论文全文数据库 医药卫生科技辑》, 15 February 2023 (2023-02-15)* |
| 陶建华 等: ""面向虚实融合的人机交互"", 《中国图像图形学报》, vol. 28, no. 6, 30 June 2023 (2023-06-30)* |
| Publication number | Publication date |
|---|---|
| CN117972395B (en) | 2024-07-09 |
| Publication | Publication Date | Title |
|---|---|---|
| US11354797B2 (en) | Method, device, and system for testing an image | |
| CN111666994A (en) | Sample image data enhancement method and device, electronic equipment and storage medium | |
| CN106485192B (en) | Training method and device of neural network for image recognition | |
| CN110046622B (en) | Targeted attack sample generation method, device, equipment and storage medium | |
| CN110070115A (en) | A kind of single pixel attack sample generating method, device, equipment and storage medium | |
| KR20220008224A (en) | Layout analysis method, reading assisting device, circuit and medium | |
| JP5261493B2 (en) | Extended image identification | |
| CN115223042A (en) | Target identification method and device based on YOLOv5 network model | |
| CN111223061A (en) | Image correction method, correction device, terminal device, and readable storage medium | |
| US11410016B2 (en) | Selective performance of deterministic computations for neural networks | |
| CN115374817A (en) | Emotion recognition method, device, equipment and storage medium based on hybrid graph network | |
| CN112561074B (en) | A machine learning interpretable method, device and storage medium | |
| CN112560864A (en) | Image semantic segmentation method and device and training method of image semantic segmentation model | |
| CN117972395B (en) | Multi-channel data processing method and device, electronic equipment and storage medium | |
| CN113299298B (en) | Residual error unit, network and target identification method, system, device and medium | |
| CN110299148B (en) | Tensorflow-based voice fusion method, electronic device and storage medium | |
| CN111435448B (en) | Image salient object detection methods, devices, equipment and media | |
| CN112613544B (en) | Target detection method, device, electronic device and computer readable medium | |
| CN114550186A (en) | Method and device for correcting document image, electronic equipment and storage medium | |
| CN115512369A (en) | Document image layout analysis method and device, computer equipment and storage medium | |
| CN115409070A (en) | Method, device and equipment for determining critical point of discrete data sequence | |
| CN114443873A (en) | Data processing method, device, server and storage medium | |
| CN111310891A (en) | Convolution operation method, device, equipment and storage medium | |
| US12026396B2 (en) | Word based channels last ordering in memory | |
| CN115456858B (en) | Image processing method, device, computer equipment and computer readable storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |