Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in the present description and the appended claims, the term "if" may be interpreted in context as "when …" or "once" or "in response to a determination" or "in response to detection. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
The following describes the technical scheme of the embodiment of the application.
With the rapid development of cognitive neuroscience, the acquisition and analysis of brain electrical signals has become one of the important technical means for psychological and behavioral research. However, due to the characteristics of complex and changeable brain electrical signals and large information redundancy, it is important to select a proper analysis method or model for the extracted features. The current psychological assessment method based on the electroencephalogram signals is mainly focused on single analysis of time domain features or frequency domain features of Event Related Potential (ERP), and the generalized feature extraction strategy for acquiring the full-segment signals through fixed parameter intervals ignores inherent differences of signal samples and influences subsequent psychological index modeling effects.
The root of the complex and changeable brain electrical signals is the interactive regulation of the mental (left brain) and intuitive (right brain) thinking modes of the human brain. Under the rational thinking, the brain electrical activity of the tested person tends to be more stable, while under the influence of the intuitive thinking, the brain electrical signal is easy to generate severe fluctuation. If the same time window is used for extracting the brain electrical characteristics of the tested person in the steady state and the excited state, it is difficult to accurately describe the two different states. Therefore, it is important to set a function that can match different rates of state change.
At present, scholars try to extract time-frequency characteristics of ERP by using adaptive filtering or wavelet transformation, but most of the scholars still process the whole signal by adopting a unified strategy, and feature transformation trends in different intervals cannot be distinguished. And a time window is designed based on comprehensive indexes such as an average value or a variance, and the subtle difference of the brain electric mode of the tested individual in a specific cognitive task is difficult to reflect. In general, the existing methods lack differential modeling of steady state and excited state electroencephalogram signals, nor do they implement targeted optimization of feature extraction strategies.
In order to solve the problems, the invention provides a visual event detection method for extracting auxiliary optimization features of a steady state. By constructing a dynamic function of a stable interval and an optimized fixed interval of a salient interval, the problem of insufficient window function design in the current method is solved, self-adaptive interval adjustment is realized, and stable characteristics of electroencephalogram signals and electroencephalogram salient characteristics corresponding to visual events in a stable state are extracted. Compared with the prior art, the method can distinguish brain electrical change modes under different brain states, and is assisted with stable characteristic setting of the optimal expression mode of the salient features, thereby providing a new idea for subsequent improvement of mental assessment based on the electroencephalogram.
The visual event refers to that the external visual stimulus or prompt causes obvious electroencephalogram response of a tested person in the process of carrying out a psychological assessment or cognitive task. Typical visual events include eye opening, blinking, turning around, etc. movements, and electroencephalogram activation caused by viewing images, text visual content. These visual events cause a prominent change in the brain electrical pattern of the subject, and if modeled using the same strategy as the steady state brain electrical signal, the cognitive or emotional changes caused by the visual events cannot be accurately detected and expressed.
Referring to fig. 1, fig. 1 is a flow chart of a method for detecting a psychological assessment visual event based on an electroencephalogram according to an embodiment of the present application. The detection method of the psychological assessment visual event based on the electroencephalogram signal can be applied to computer equipment, wherein the computer equipment comprises, but is not limited to, intelligent mobile phones, notebook computers, tablet computers, desktop computers, physical servers, cloud servers and the like. As shown in fig. 1, the method for detecting a psychological evaluation visual event based on an electroencephalogram according to the present embodiment includes steps S101 to S107, which are described in detail below:
step S101, acquiring brain wave images of a user at a plurality of moments, and forming an original brain wave time series signal according to the plurality of brain wave images.
Specifically, by collecting brain wave images of a user as a subject at different times (the subject and the user are referred to as the same subject throughout the text), an original brain wave time-series signal is constituted. An electroencephalogram acquisition device is adopted to acquire electroencephalogram signals of a tested person when a visual task is executed, the electroencephalogram signals are converted into a series of electroencephalogram image sequences, each electroencephalogram image reflects the distribution condition of the brain on the electric activity of a specific time point, and the acquisition frequency of the electroencephalogram images is set to be 500Hz. Let the acquired electroencephalogram image sequence be represented as { I (t) }, t=1, 2..n, where t represents the time point and N is the total time point.
In order to construct an original brain wave time series signal X (t), each two-dimensional image I (t) needs to be converted into a one-dimensional vector, and the method is characterized in that all elements of each image matrix are unfolded and spliced according to rows to form a long vector, and then all vectors are arranged in time sequence to form a matrix X, wherein the t-th row vector X (t) corresponds to the brain wave image I (t) at the moment t. The conversion process can be expressed as X (t,:) =f (I (t)), where f is a function of mapping the image I (t) to a one-dimensional vector, and all elements of the image matrix are stitched into one long vector by line expansion in a matrix-straightened manner. In this way, the original two-dimensional electroencephalogram image sequence is converted into a one-dimensional time-series signal X (t).
Step S102, first reference characteristic signals of every two adjacent brain wave images in the original brain wave time sequence signals are obtained, second reference characteristic signals corresponding to every two adjacent first reference characteristic signals are obtained, and third reference characteristic signals corresponding to every two adjacent second reference characteristic signals are obtained.
Specifically, a first energy variation between adjacent images is extracted as a first reference feature signal, an energy variation between adjacent first reference feature signals is extracted as a second reference feature signal, and an energy variation between adjacent second reference feature signals is extracted as a third reference feature signal. Meanwhile, if the accuracy of the subsequent model training is improved, the fourth … … reference characteristic signals and the fifth … … reference characteristic signals can be continuously extracted by the same method.
In some embodiments, the acquiring the first reference feature signals of each two adjacent brain wave images in the original brain wave time series signal, acquiring the second reference feature signals corresponding to each two adjacent first reference feature signals, and acquiring the third reference feature signals corresponding to each two adjacent second reference feature signals includes: extracting first energy changes corresponding to every two adjacent brain wave images, wherein the first energy changes are used as the first reference characteristic signals; extracting second energy changes corresponding to every two adjacent first reference characteristic signals, wherein the second energy changes are used as the second reference characteristic signals; and extracting third energy changes corresponding to every two adjacent second reference characteristic signals, wherein the third energy changes are used as the third reference characteristic signals.
Next, energy variation between adjacent electroencephalograms needs to be extracted as the first reference characteristic signal E1 (t). The specific calculation formula is as follows: e1 (t) = ||x (t,: -X (t+1,): i, where i represents the Frobenius norm of the matrix, i.e., the sum of squares of all elements plus the square root. The Euclidean distance is adopted as the distance measurement between the images, so that the difference degree of the two images on the pixel value can be reflected better, when the two images are completely identical, the distance is 0, and the larger the difference is, the larger the distance is. Through the above calculation, the first reference characteristic signal E1 (t) is obtained, which reflects the degree of energy variation between the electroencephalogram images at adjacent timings.
After the E1 (t) sequence is obtained, the energy variation between adjacent values in the E1 (t) sequence is further extracted as a second reference characteristic signal E2 (t). The calculation formula is E2 (t) = ||E1 (t) -E1 (t+1) |, and Euclidean distance is still used as distance measurement between adjacent first reference features. E2 The (t) sequence can further describe the change trend of the electroencephalogram signal, reflects the change condition of energy change at adjacent moments, and if the E2 (t) value is smaller, the energy change is stable at the adjacent moments, otherwise, if the E2 (t) value is larger, the energy change is severely fluctuated.
Similarly to the extraction of E2 (t), a third reference characteristic signal E3 (t) is also extracted for measuring the energy variation between adjacent second reference characteristic signals E2. The calculation formula is E3 (t) = ||E2 (t) -E2 (t+1) |, and Euclidean distance is also adopted as a measurement. E3 The (t) sequence reflects the change condition of the change rate of the tested brain electrical energy change, and can further capture the high-order detail characteristics of the brain electrical signal change.
In order to eliminate the influence of noise interference, smoothing such as moving average filtering may be performed on each of three sequences E1 (t), E2 (t), and E3 (t). The method comprises the steps that each value in a sequence is replaced by an arithmetic average value of a plurality of adjacent values before and after the value, the window size of a moving average can be set according to actual requirements, and 5-11 points are usually taken. The smoothed E1' (t), E2' (t) and E3' (t) sequences respectively replace the original sequences to participate in subsequent calculations.
Step S103, calculating a first difference corresponding to the first reference feature signal and the second reference feature signal and a second difference corresponding to the second reference feature signal and the third reference feature signal at the same time.
Specifically, a first difference signal D1 (t) is calculated, which reflects the degree of difference between the first reference characteristic signal E1 (t) and the second reference characteristic signal E2 (t), and an absolute difference value is adopted as a difference metric, wherein a specific calculation formula is D1 (t) = |E1 (t) -E2 (t) |. Here, || denotes an absolute value sign. When the absolute difference between E1 (t) and E2 (t) is larger, the larger the deviation between the first reference characteristic signal E1 and the second reference characteristic signal E2 at the moment t is, larger signal fluctuation is reflected, otherwise, when the absolute difference is smaller, the two reference characteristic signals are closer at the moment, and the signal fluctuation is relatively gentle.
Next, a second difference signal D2 (t) is calculated, which is used to reflect the degree of difference between the second reference characteristic signal E2 (t) and the third reference characteristic signal E3 (t), and the calculation formula is D2 (t) = |e2 (t) -E3 (t) |, and the absolute difference value is also used as the difference measure. A larger value of D2 (t) indicates that E2 (t) and E3 (t) differ significantly at that time, indicating that the signal has changed drastically at that time, and a smaller value of D2 (t) indicates that the signal is relatively stationary. Because the E3 (t) sequence reflects the change of the energy change rate, the D2 (t) can further capture the change condition of the high-order instantaneous characteristic of the signal, the corresponding physical meaning is the change quantity of the second derivative of the signal, and the mutation point of the signal can be detected more acutely.
To help understand the calculation procedure of this step, a specific numerical example is given in which, assuming that three reference feature sequences of E1, E2, and E3 are obtained in the above step, and at time point t=100, the values of the three sequences are E1 (100) =2.5, E2 (100) =1.8, and E3 (100) =3.2, respectively, D1 (100) = |e1 (100) -E2 (100) |= |2.5-1.8|=0.7, and D2 (100) = |e2 (100) -E3 (100) |= |1.8-3.2|= 1.4, respectively. It can be seen that, at the time t=100, the D1 value is smaller and the D2 value is larger, which indicates that there is a larger deviation between the energy change of the electroencephalogram signal and the rate of change of the energy change, and the second derivative change of the signal at the time is also larger, which may be caused by sudden visual event excitation, and the signal segment near the time needs to be subjected to key analysis and processing in a subsequent step.
For the two difference signal sequences D1 (t) and D2 (t) obtained by calculation, smoothing such as moving average filtering is required to reduce the influence of noise. In particular, moving average filtering is used, the smoothing principle of which is to replace each value in the sequence with an arithmetic average of several adjacent values before and after the value. Assuming that the smoothing window size is L (L is a positive odd number), the calculation formulas of D1 '(t) and D2' (t) after smoothing are as follows:
d1 '(t) = (1/L) ×Σ (i=t-L/2 to t+l/2) D1 (i) d2' (t) = (1/L) = Σ (i=t-L/2 to t+l/2) D2 (i)
Where Σ represents the summation symbol. In the above formula, the integer part is taken by L/2, the window size L usually takes small odd values such as 5, 7 or 9, and the specific value needs to be adjusted by combining with the statistical characteristics of actual data. By the smoothing processing, the high-frequency noise components in the D1 (t) and D2 (t) sequences can be effectively reduced, and smoother differential signal sequences can be obtained.
Through the above calculation and smoothing steps, two sequences of smoothed difference signals D1 '(t) and D2' (t) are finally obtained, which reflect the degree of difference between the electroencephalogram signal and its first and second derivatives, respectively. In general, if D1 '(t) and D2' (t) take larger values at the same time, this indicates that the electroencephalogram signal experiences severe fluctuations and may correspond to an prominent response triggered by a visual event, whereas if two difference signals take smaller values at the same time, this indicates that the signal is relatively stationary and may correspond to a baseline state being tested.
Step S104, respectively extracting a first stable signal set and a second stable signal set from the original brain wave time series signals according to the first difference and the second difference.
Specifically, the difference is compared to determine a first stable section and a second stable section of the brain wave signal, and brain wave images in the stable sections are extracted as a first stable signal set and a second stable signal set.
In some embodiments, the extracting a first stationary signal set and a second stationary signal set from the original brain wave time series signal according to the first difference and the second difference, respectively, comprises: acquiring a first stable interval corresponding to the first difference; acquiring a second stable interval corresponding to the second difference; extracting a first stationary signal set from the original brain wave time series signal according to the first stationary interval; and extracting a second stable signal set from the original brain wave time series signal according to the second stable interval.
First, two thresholds θ1 and θ2 are set for determining the magnitudes of the sequence values of D1 '(t) and D2' (t), thereby distinguishing the steady state from the excited state. When D1 '(t) < θ1 and D2' (t) < θ2, the signal corresponding to the time t is considered to be in a stationary state, whereas when D1 '(t) > θ1 or D2' (t) > θ2, the signal corresponding to the time t is considered to be in an excited state. The setting of the thresholds theta 1 and theta 2 needs to be optimized by combining with the statistical characteristics of actual data, and the average value of the sequences D1 '(t) and D2' (t) can be generally taken as an initial threshold by adding a standard deviation of a certain multiple, so that fine adjustment is performed according to the classification effect.
Then, the original electroencephalogram signal is segmented based on the threshold judgment, specifically, each time t is scanned sequentially from the starting point of a time sequence, if a plurality of continuous time points simultaneously meet D1 '(t) < theta 1 and D2' (t) < theta 2, signal segments corresponding to the time points are determined to be stable sections, otherwise, if any time point D1 '(t) > or theta 1 or D2' (t) > or theta 2 exists, the signal segments corresponding to the time points are determined to be prominent sections. In this way, the entire time series can be divided into several stationary and protruding sections that do not overlap each other. It should be noted that, to avoid excessive subdivision, a minimum duration of the stationary section and the protruding section may be set in actual operation, for example, 10 time points, and a cell below the duration may be incorporated into an adjacent larger section.
Next, from all the stationary intervals determined in the previous step, the corresponding brain wave image sequences are extracted, respectively, and a first stationary signal set R1 and a second stationary signal set R2 are constructed. The specific method is that for each stable interval [ t_start, t_end ], a subsequence X (t_start: t_end) is extracted from an original brain wave time sequence X (t), and a two-dimensional image sequence { I (t) } corresponding to the subsequence X (t_start: t_end, t=t_start, & gt, t_end is spliced together to be used as one element of R1 or R2. In this way, two stationary signal sets R1 and R2 are obtained, which respectively contain the corresponding electroencephalographic sequences when the test is in a rational mental state and an intuitive mental state in the visual task.
To help understand the implementation process of this step, a specific example is given in which, assuming that two smoothed difference signal sequences of D1 '(t) and D2' (t) are obtained in the above step, 1.5 times the mean value of the D1 '(t) sequence is taken as a threshold θ1=1.2, and 2 times the mean value of the D2' (t) sequence is taken as a threshold θ2=0.8 through statistical analysis. Starting from the time point t=1, the values of D1 '(t) and D2' (t) are scanned sequentially, and if D1 '(t) <1.2 and D2' (t) <0.8 are satisfied simultaneously for a plurality of consecutive times, the signal segment corresponding to these times is determined as a plateau section, otherwise, is determined as a salient section. Assuming that, through scanning, both D1 '(t) and D2' (t) continuously satisfy the threshold condition in the time periods [50,150] and [200,300], the signal segments X (50:150) and X (200:300) corresponding to the two time periods are determined as stationary intervals, and the corresponding two-dimensional image sequences { I (t) }, t=50,..150 and { I (t) }, t=200,..300 are extracted as two elements in R1 and R2, respectively.
Through the segmentation and extraction processes, the original electroencephalogram signal is successfully divided into a stable section and a prominent section, and a first stable signal set R1 and a second stable signal set R2 are extracted from the stable section. The two stationary signal sets reflect the brain electrical patterns tested in different cognitive states, R1 corresponds to the brain electrical image sequence at rational thinking and R2 corresponds to the brain electrical image sequence at intuitive thinking. Meanwhile, the signal segments corresponding to the remaining salient regions may include salient response features triggered by visual events.
Step S105, removing the first stable signal set and the second stable signal set from the original brain wave time series signal to obtain a first characteristic signal set and a second characteristic signal set.
Specifically, X (t) has been divided into a number of stationary intervals and protruding intervals in the above steps, and the first stationary signal set R1 and the second stationary signal set R2 are extracted from the stationary intervals. It is now necessary to extract from the remaining salient intervals a corresponding first set of characteristic signals T1 and a second set of characteristic signals T2 reflecting the electroencephalographic salient response of the tested to the visual event in the rational and intuitive thinking state of the visual task.
To obtain T1 and T2, a step of first recalling the segmentation results in the above step, locating the start-stop time points of all protruding sections from X (T), denoted as { [ t1_start, t1_end ], [ t2_start, t2_end ], [ tm_start, tm_end ] }. Wherein M is the total number of protruding sections. Then for each protruding section, the corresponding signal segment is extracted from X (T), and the elements in T1 and T2 are respectively constructed. The specific method is that for the ith salient interval [ ti_start, ti_end ], an X (ti_start: ti_end) subsequence is extracted, and a corresponding two-dimensional image sequence { I (T) }, t=ti_start, & gt, ti_end is taken as a new element in T1 or T2. The determination of which category (rational or intuitive state) the salient interval belongs to can be made based on the segmentation thresholds θ1 and θ2 and the values of the difference signals D1 '(t) and D2' (t). If any time T is in [ ti_start, ti_end ] so that D1 '(T) is more than or equal to theta 1, adding a signal segment corresponding to the interval into T1, otherwise, if any time T is in the range of D2' (T) is more than or equal to theta 2, adding the signal segment into T2. In this way, the signal segments corresponding to all the protruding sections can be successfully divided into T1 and T2.
It should be noted that, in order to obtain a more robust set of T1 and T2, in actual operation, the start and stop time points of each protruding section may be extended appropriately, i.e. a few time points are extended to two sides on the basis of [ ti_start, ti_end ]. This ensures that the salient responses are contained entirely without losing critical information due to boundary truncation. The extended time length can be set according to practical situations, and usually 5-20 time points are taken. In addition, to avoid over-subdivision, a minimum duration threshold for the highlighted interval may be set, such as a cell below the threshold (e.g., 10 time points), and then eliminated or incorporated into an adjacent larger interval.
Assuming that the above-described processing is performed, t1= { S11, S12..s 1P } and t2= { S21, S22,.. S2Q, where P and Q are the number of elements (i.e., image sequences) in T1 and T2, respectively. It can be seen that the elements in T1 correspond to the electroencephalographic salient response to a visual event being tested in a mental state, while the elements in T2 correspond to the salient response in an intuitive state. From the construction point of view, both T1 and T2 are extracted from the remainder of the X (T) after the stationary signal portion is removed, so they can be considered to contain key electroencephalogram information for visual event excitation.
To facilitate subsequent feature extraction and classification, the image sequences in T1 and T2 need to be uniformly processed, for example, by interpolation to have each sequence the same length, or by Dynamic Time Warping (DTW) to align different sequences in time, etc. The processing mode can be determined according to specific task requirements, and the purpose is to keep all elements in T1 and T2 consistent in dimension and time scale, so that calculation of feature vectors and training of models are facilitated. It is assumed that, after processing, each element of T1 and T2 is converted into a one-dimensional vector of length N, i.e., t1= { V11, V12,..v. V1P }, t2= { V21, V22..v. V2Q }, wherein each Vij (i=1, 2; j=1..p or Q) is a column vector of length N.
Finally, all vectors in T1 and T2 are spliced in sequence to obtain a first characteristic signal set F1 and a second characteristic signal set F2, which reflect all brain electrical responses of the tested to the visual event in different cognitive states. The concrete steps are as follows:
F1 = [ V11, V12, ], V1P ] (p×n matrix) f2= [ V21, V22, ], V2Q ] (q×n matrix).
Wherein each row of F1 is a feature vector, P rows are shared, and each row of F2 is a feature vector, Q rows are shared. It is worth noting that since F1 and F2 are prominent responsive portions directly from the original electroencephalogram signal X (t), they can be considered to contain the most critical electroencephalogram characteristic information for visual event excitation.
The whole implementation procedure of the above steps is illustrated assuming that the original brain wave time series X (t) has been divided into a plurality of stationary intervals and protruding intervals in the above steps, and two stationary signal sets of R1 and R2 are extracted from the stationary intervals. T1 and T2 now need to be taken from the remaining protruding intervals. Based on the segmentation results of the above steps, it is assumed that a total of 5 protruding intervals are detected, in the time ranges of [100,200], [250,320], [380,420], [500,570], and [620,680], respectively. The 5 signal segments, namely X (100:200), X (250:320), X (380:420), X (500:570) and X (620:680), are first extracted from X (t).
Next, it is necessary to determine what kind of cognitive state (rational thinking or intuitive thinking) each signal segment corresponds to, so that it is added to T1 or T2. Assuming that there is some time T such that the difference signal D1' (T) is ≡θ1 (threshold θ1=1.2 set by the above steps) within the period [100,200], the image sequence { I (T) } corresponding to X (100:200, t=100..200 is added to T1. Similarly, if there is some time T such that D2' (T) is ≡θ2 (θ2=0.8 set by the above steps) within the period [250,320], { I (T) }, t=250..320 is added to T2. And so on, the initial element sets of T1 and T2 are ultimately available.
To ensure that the salient responses are completely contained, a suitable extension may be made for each time period, for example, 10 time points are extended to two sides on the basis of [100,200], i.e., the image sequence corresponding to [90,210] is taken as a new element of T1. Furthermore, if it is detected that the protruding section duration is below a set threshold (e.g., 10 time points), then it is culled or merged into an adjacent larger section. After these processes, it is assumed that t1= { S11, S12, S13, S14} and t2= { S21, S22, S23} are obtained, where each Sij is an image sequence.
The image sequences in T1 and T2 are then subjected to the necessary unified processing, such as by interpolation to make each sequence 500 in length (i.e., n=500), to obtain t1= { V11, V12, V13, V14} and t2= { V21, V22, V23} where Vij is a 500-dimensional column vector. And finally, all vectors in the T1 and the T2 are spliced in sequence, so that a first characteristic signal set F1 (4 multiplied by 500 matrix) and a second characteristic signal set F2 (3 multiplied by 500 matrix) can be obtained.
And S106, carrying out feature fusion on the first stable signal set, the second stable signal set, the first feature signal set and the second feature signal set to obtain a composite feature sample.
Specifically, the key to follow is to effectively fuse these four classes of features. Since the stationary signal features and the visual event features respectively reflect the brain electrical patterns tested in different cognitive states, combining them together helps to improve the discrimination capability of subsequent classification. The specific fusion strategy needs to weigh the contribution degree of various characteristics to classification and determine a reasonable fusion weight. The application adopts a self-adaptive weighting method based on the sample cohesion degree, so that the weight value can be automatically matched with the internal distribution characteristics of the data.
In some embodiments, the feature fusing the first stationary signal set, the second stationary signal set, the first feature signal set, and the second feature signal set includes: respectively acquiring a first stable signal and a second stable signal corresponding to the first stable signal set and the second stable signal set, and respectively acquiring a first visual event feature and a second visual event feature corresponding to the first characteristic signal set and the second characteristic signal set; and carrying out feature fusion on the first stable signal, the second stable signal, the first visual event feature and the second visual event feature to obtain the composite feature sample.
Exemplary, the acquiring the first stationary signal and the second stationary signal corresponding to the first stationary signal set and the second stationary signal set respectively includes: respectively acquiring a first dynamic window function and a second dynamic window function corresponding to the first stable signal set and the second stable signal set; extracting a first stationary signal from the first stationary signal set according to the first dynamic window function, and extracting a second stationary signal from the second stationary signal set according to the second dynamic window function
It should be noted that, in some embodiments, the acquiring the first visual event feature and the second visual event feature corresponding to the first feature signal set and the second feature signal set respectively includes: respectively calculating energy characteristics and stability characteristics of the first stable signal set, the second stable signal set, the first characteristic signal set and the second characteristic signal set, and generating adjustment coefficients according to a plurality of energy characteristics and stability characteristics; respectively optimizing the first dynamic window function and the second dynamic window function according to the adjustment coefficient to obtain a first optimization window and a second optimization window; and extracting a first visual event feature from the first characteristic signal set according to the first optimizing window, and extracting a second visual event feature from the second characteristic signal set according to the second optimizing window.
For the first stable signal set and the second stable signal set, analyzing the statistical characteristics of the internal signal change rates of the first stable signal set and the second stable signal set, and adaptively determining a first dynamic window function and a second dynamic window function according to the change rates; and then, respectively utilizing the first dynamic window function and the second dynamic window function to extract the characteristics of the two stable signals. ( Description: although stationary signals fluctuate less than prominent signals, there is still some pattern of subtle variations within them that might mask or average if feature extraction is performed using a fixed window, thereby affecting the accuracy of subsequent classification. The size and the shape of the window can be adaptively adjusted according to the local change condition of the signal by the dynamic window function, so that the window can be closely matched with the change trend of the signal, thereby capturing the tiny change in the stable signal better and improving the effect of feature extraction. )
The following describes, in an embodiment, a specific embodiment corresponding to the above description:
First, each element of R1 and R2 (i.e., the stationary electroencephalogram sequence) needs to be transformed to extract its first derivative sequence D1 '(t) and second derivative sequence D2' (t). The two derivative sequences reflect the rate of change of the stationary signal over different orders, with larger derivative values corresponding to regions of more intense variation. Next, statistics are calculated for each of D1 '(t) and D2' (t), including matrix norms such as mean μ, standard deviation σ, kurtosis κ, and skewness γ. The statistics can better characterize the distribution characteristics of the change rate of the signal on different orders, such as the mean value reflects the overall change degree, the standard deviation reflects the fluctuation range, the kurtosis characterizes the mutation frequency, the skewness reflects the mutation directionality and the like. Taking D1 '(t) as an example, let its mean be μ1, standard deviation be σ1, kurtosis be κ1, skewness be γ1, and similarly, statistics corresponding to D2' (t) be μ2, σ2, κ2, γ2.
With the statistics described above, the present application can design adaptive dynamic window functions W1 (t) and W2 (t) for extracting the features of R1 and R2. Wherein W1 (t) is for R1 and W2 (t) is for R2. The design of the dynamic window function is based on the following principles that 1) the window size can be adaptively adjusted to match the change rate of the signal at the current moment, 2) a larger change rate corresponds to a smaller window, a smaller change rate corresponds to a larger window, and 3) a smooth window (such as a Gaussian window) is adopted near the mutation point to treat, so that excessive transition change is avoided.
Specifically, W1 (t) and W2 (t) may be generated by the following parameterized functions:
W1(t) = a1exp(-b1|D1'(t)|) + c1exp(-d1|D2'(t)|) + f1gausswin(L1) ；
W2(t) =a2exp(-b2*|D1'(t)|) + c2exp(-d2|D2'(t)|) + f2gausswin(L2)；
The adaptive parameters of a1, b1, c1, d1, f1, a2, b2, c2, d2, f2 and f2 are W1 (t) and W2 (t) and are obtained through estimation by an optimization algorithm, gauss window (L) is a standard Gaussian window function, the window length L takes a value of 5-11, and I represents an absolute value sign. It can be seen that the dynamic windows W1 (t) and W2 (t) are composed of three parts, wherein the first two parts are exponential functions, the power exponent of the first two parts is determined by the absolute values of D1 '(t) and D2' (t), the larger the change rate is, the faster the window decays, the smaller the window range is, and the third part is a smooth Gaussian window used for processing the transition region near the mutation point so as to avoid the abrupt transition process. The linear superposition of the three components ensures that the whole window shape can be better matched with the local variation characteristic of the signal.
In order to obtain the optimal parameter combinations a1, b1, c1, d1, f1 and a2, b2, c2, d2, f2, f2, the application needs to design an objective function and adopts an optimization algorithm (such as simulated annealing, particle swarm and the like) to perform parameter estimation. The design principle of the objective function is that the feature vector sequences V1 (t) and V2 (t) extracted through the dynamic windows W1 (t) and W2 (t) have the largest cohesive degree, namely the similarity between the feature vectors of the same class is the highest. The specific method comprises the following steps:
1) initializing parameters a1, b1, c1, d1, f1 and a2, b2, c2, d2, f2, f2 to small positive values, 2) extracting a feature vector sequence V1 (t) for each plateau sequence in R1 based on the current W1 (t), 3) extracting a feature vector sequence V2 (t) for each plateau sequence in R2 based on the current W2 (t), 4) calculating cohesive degree scores S1 and S2 of V1 (t) and V2 (t) as optimized objective function values, 5) optimizing the parameters by iteration using simulated annealing or the like to maximize the parameters S1 and S2, 6) outputting optimal W1 (t) and W2 (t) parameter combinations when the S1 and S2 converge.
The specific calculation formulas of the cohesive degree scores S1 and S2 are as follows:
s1=Σ (k=1 to K1) (1/N1K) ×Σ (p=1 to N1K, q=1 to N1K) exp (- |v1kp-v1kq||ζ2/2σ 1^2);
S2=Σ (k=1 to K2) (1/N2K) ×Σ (p=1 to N2K, q=1 to N2K) exp (- |v2kp-v2kq||ζ2/2σ 2^2);
Wherein K1 and K2 are the number of elements (stationary sequences) in R1 and R2, N1K and N2K are the lengths of the kth R1 and R2 sequences, V1kp and V2kp are the p-th feature vectors extracted from the kth R1 and R2 sequences using W1 (t) and W2 (t), respectively, and sigma 1 and sigma 2 are the average Euclidean distances of the feature vectors in R1 and R2, respectively, for normalization processing. It can be seen that S1 and S2 are in fact intra-class compactions summing all feature vectors in R1 and R2, the larger the value the more concentrated the vectors of the same class, the better the cohesiveness of the feature set. Therefore, by maximizing S1 and S2, the dynamic windows W1 (t) and W2 (t) can capture the most representative local variation features in R1 and R2, respectively, thereby improving the accuracy of the subsequent classification.
And finally obtaining self-adaptive dynamic windows W1 (t) and W2 (t) through the parameter optimization flow. And then respectively acting on each stable sequence in R1 and R2 to extract corresponding characteristic vector sequences, wherein the characteristic vector sequences are a first stable signal characteristic set and a second stable signal characteristic set. The specific method comprises the steps of calculating convolution of W1 (t) and I1 (t) for each stable electroencephalogram sequence I1 (t) in R1 to obtain a corresponding characteristic sequence F1 (t). And all F1 (t) are spliced to obtain a first stable signal characteristic set FP1, and the dimension is K1× (N1-L1+1). Where K1 is the number of elements (plateau sequences) in R1, N1 is the length of each plateau sequence, and L1 is the window length of W1.
And similarly, calculating convolution of the sequences I2 (t) in the R2 by W2 (t) and I2 (t) to obtain a characteristic sequence F2 (t). And all F2 (t) are spliced to obtain a second stable signal characteristic set FP2, and the dimension is K2× (N2-L2+1). Here, K2, N2, L2 correspond to the number of elements in R2, the sequence length, and the window length of W2, respectively. Where x represents the convolution operation.
By means of the window convolution, the first stable signal characteristic and the second stable signal characteristic are successfully extracted from the R1 and the R2 respectively, the characteristics not only keep the basic mode of the electroencephalogram signal in a stable state, but also capture the inherent tiny change of the signal, and effective characteristic support is provided for subsequent visual event classification.
Meanwhile, calculating the statistical difference of two types of signals (a first stable signal set and a second stable signal set, a first characteristic signal set and a second characteristic signal set) on a first energy characteristic, a first stability index, a second energy characteristic and a second stability index;
The four indexes have the physical significance that the first energy characteristic reflects the overall energy level of the signal and can be used for distinguishing a high energy state from a low energy state, the first stability index characterizes the stability degree of signal fluctuation, the smaller the value is, the more stable the signal is, the second energy characteristic is a local energy index and can capture the instantaneous energy change of the signal, and the second stability index reflects the change condition of the second derivative (acceleration) of the signal and can be used for identifying the mutation point. By comparing the statistical differences of the different signal sets on the four indexes, the respective characteristic modes can be found, so that the subsequent window optimization strategy is guided.
Firstly, the four statistical indexes are respectively extracted from R1, R2, F1 and F2. The specific method comprises the following steps: for the first stationary signal set R1, it is assumed to contain K1 elements (stationary electroencephalogram sequence), denoted { V11, V12., V1K1}, where each Vij (j=1., K1) is a row vector of length N. The first energy characteristic of R1 may be defined as the sum of the two norms of all vectors, namely: e1_r1= Σ (j=1) to K1) Vij; wherein I representation of the two norms of the vector. The first stability index may be described by the variance of the vector sequence, the calculation formula s1_r1=Σ (j=1 to K1) VAR (Vij)/K1; here VAR (-) is a function that calculates the variance of the vector, i.e. the variance is calculated for all elements in the vector. The second energy feature is defined as the mean of all vector second-order norms, and can characterize the instantaneous energy level of the element sequence in R1, E2_R1=Σ (j=1 to K1) ||Vij' |ζ2/K1; where Vij' is the first derivative of Vij, and 2 represents the squared norms. The second stability index can reflect the mutation frequency of the sequence in R1 by using the acceleration norm average description of the vector sequence, wherein S2_R1=Σ (j=1 to K1) ||Vij '' |/K1; where Vij "is the second derivative of Vij, i are the two norms of the vector.
In a similar manner to that described above, four indexes E1_R2, S1_R2, E2_R2, S2_R2 and E1_F1 corresponding to R2, F1 and F2 can be calculated s1_f1, e2_f1, s2_f1, and e1_f2 s1_f2, e2_f2, s2_f2. It should be noted that, for the feature vector sequences in F1 and F2, since they may have different lengths due to different events, interpolation processing may be performed on each sequence to have the same length N when calculating the above index.
After the values of R1, R2, F1 and F2 on the four indexes are taken, the application can calculate the statistical difference between the four indexes and can be used as the basis for determining the adjustment coefficient. Specifically, the following 6 difference values are calculated using a pairwise comparison strategy:
diff_E1 = |E1_R1 - E1_F1| + |E1_R2 - E1_F2|；
diff_S1 = |S1_R1 - S1_F1| + |S1_R2 - S1_F2|；
diff_E2 = |E2_R1 - E2_F1| + |E2_R2 - E2_F2|；
diff_S2 = |S2_R1 - S2_F1| + |S2_R2 - S2_F2|；
diff_EP = |E1_R1 - E1_R2| + |E1_F1 - E1_F2|；
diff_SP = |S1_R1 - S1_R2| + |S1_F1 - S1_F2|；
Wherein diff_E1 and diff_S1 are the differences between the stationary signal and the characteristic signal of the first energy characteristic and the first stability index, respectively, diff_E2 and diff_S2 are the differences between the two types of signals of the second energy characteristic and the second stability index, and diff_EP and diff_SP reflect the differences between the stationary signal and the characteristic signal of the first energy characteristic and the first stability index, respectively. The magnitude of the 6 difference values can better characterize the difference between the stationary signal and the characteristic signal on the four statistical indexes.
Next, these 6 difference values are normalized as weights for determining the first adjustment coefficient α and the second adjustment coefficient β:
w1 = diff_E1 / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
w2 = diff_S1 / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
w3 = diff_E2 / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
w4 = diff_S2 / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
w5 = diff_EP / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
w6 = diff_SP / (diff_E1 + diff_S1 + diff_E2 + diff_S2 + diff_EP + diff_SP)；
wherein w 1-w 6 are normalized weights corresponding to 6 difference values, and has w1+w2+w3 +w2 +w3. These 6 weights reflect the relative importance of the internal/external differences of the four statistical indicators and plateau/feature signals to window adjustment.
With the weights, a first adjustment coefficient alpha and a second adjustment coefficient beta can be generated:
α = w1E1_R1/E1_F1 + w2S1_R1/S1_F1 + w3E2_R1/E2_F1 + w4S2_R1/S2_F1 + w5E1_R1/E1_R2 + w6S1_R1/S1_R2 ；
β = w1E1_R2/E1_F2 + w2S1_R2/S1_F2 + w3E2_R2/E2_F2 + w4S2_R2/S2_F2 + w5E1_R2/E1_R1 + w6S1_R2/S1_R1；
It can be seen that α and β are weighted ratios of four statistical indexes between the stationary signal and the characteristic signal, and the difference information between the interior of the stationary signal and the interior of the characteristic signal is fused. The magnitude of these two adjustment coefficients reflects the extent to which the window function subsequently needs to be stretched or compressed. If alpha >1, the first optimization window is expanded to capture more stationary signal features, whereas if alpha <1, the window is contracted to suppress some redundancy, and similarly beta adjusts the second optimization window to follow a similar strategy.
The first and second optimization windows are generated by multiplying the first and second adjustment coefficients, respectively, with a preset initial window size, specifically, the initial window size W0 needs to be determined first, which is an empirical value, typically depending on the acquisition frequency of the analyzed electroencephalogram sequence and the minimum time scale features that need to be captured. Generally, the value of W0 ranges from 5 to 25 time points, the smaller the value is, the more detail features can be captured, but the more detail features are more easily interfered by noise, and the larger the opposite value is, the more stable overall features can be extracted, but some transient change information can be lost. Therefore, the specific value of W0 needs to be adjusted according to the requirements of the actual task, and can also be adjusted by a cross-validation method and the like.
Assume that the initial window size W0 has been determined, for example, set to 15 time points. Then, according to the two adjustment coefficients α and β obtained in the above step, a first optimization window W1 and a second optimization window w2:w1=α×w0w2=β×w0 can be generated.
Wherein when a >1, the size of W1 will be larger than the initial window W0, which means that the window range needs to be enlarged to capture more stationary signal features, whereas when a <1, the range of W1 will be smaller than W0, so that the compression window suppresses some redundancy. Similarly, the magnitude of β will also determine the degree of stretching or compression of W2 relative to W0. Through the self-adaptive adjustment strategy, W1 and W2 can be respectively matched with the internal change characteristics of the first stable signal set R1 and the second stable signal set R2, and the effect of subsequent characteristic extraction is improved.
It should be noted that, to ensure the rationality of W1 and W2, some limitation may be imposed on the value ranges thereof. For example, a maximum window length Wmax is set, and if the calculated W1 or W2 exceeds the length, it is made equal to Wmax to avoid excessive stretching, and a minimum window length Wmin is set, and if W1 or W2 is smaller than the value, it is made equal to Wmin to prevent information loss due to excessive compression. The specific values of Wmax and Wmin also need to be tuned according to the actual task, typically Wmax is between 50100 points and Wmin is between 515 points.
Extracting a first visual event feature and a second visual event feature from the first characteristic signal set F1 and the second characteristic signal set F2 by adopting a first optimization window and a second optimization window respectively;
Specifically, F1 reflects the electroencephalogram response to a visual event in a mental state under test, and F2 corresponds to the response in an intuitive state. The present application contemplates distinguishing between the two different cognitive states by capturing the most representative visual event features from F1 and F2 by W1 and W2, respectively.
First, the constitution of F1 and F2 will be reviewed. In the step, extracting a salient response part of the original brain wave time sequence X (t) except for a stable section, and respectively constructing two matrixes F1 and F2, wherein F1 comprises P rows of feature vectors, and each row comprises an N-dimensional vector V1j (j=1.,. The number of the P); f2 contains Q rows of N-dimensional feature vectors V2j (j=1..q.). These feature vectors may be from the excitation of different visual events or may not be exactly the same length. In order to facilitate the window operation, the application needs to normalize all the vectors of F1 and F2 so that the vectors have the same dimension N. The original sequence can be subjected to equal length stretching or compression by adopting a direct interpolation method or a Dynamic Time Warping (DTW) method and the like.
Suppose that after unified processing, F1 is converted into a P×M matrix and F2 is converted into a Q×M matrix, where M is the new feature dimension and M.gtoreq.N. The first and second optimization windows W1 and W2 may then be used to filter and extract features from F1 and F2. The specific approach is to calculate its convolution with W1 for each row of vector V1j in F1 (j=1,., P):
C1j=v1 j is W1. Wherein, represents one-dimensional convolution operation, C1j is the convolution output vector, the length is M-L+1, and L is the window length of W1. This corresponds to sliding over V1j using W1 as a filter and computing the inner product point by point, resulting in a new feature sequence. Convolution operations can capture local features well on different time scales, and the length of W1 determines the degree of refinement of the acquired features. By performing a similar convolution on all vectors in F1, P convolved output sequences are obtained, forming a matrix FE1 of dimension (p× (M-l+1)) denoted as fe1= [ C11; C12; C1P ].
Each row of vectors of FE1 is a new feature sequence obtained by convolving V1j with W1, and can be regarded as a first set of visual event features extracted from F1.
For the feature vector sequence { V2j } in F2, the convolution and feature extraction may also be similarly performed using a second optimization window W2, c2j=v2j×w2; FE 2= [ C21; C22; C2Q ].
Where W2 is given a length of K, then FE2 is a (Q× (M-K+1)) dimensional matrix, and each row of vectors is a corresponding visual event feature extracted from F2. FE2 can be considered a second set of visual event features. Note that, since the lengths of W1 and W2 are generally different (i.e., l+.k), the row vector lengths of FE1 and FE2 are also different. However, in the subsequent fusion, the column vectors can be subjected to equal-length processing.
Through the filtering and convolution calculation, key electroencephalogram characteristics FE1 and FE2 of visual events under corresponding rational thinking and intuitive thinking states are successfully extracted from F1 and F2 respectively. It is worth mentioning that, since both W1 and W2 are generated by the adaptive optimization strategy in the above steps, the features contained in FE1 and FE2 can be well matched with the intrinsic variation patterns of the electroencephalogram signals in different cognitive states, so as to improve the subsequent event classification and detection capability.
To aid in understanding the above process, a specific example is given in which, assuming that after unified processing, F1 is a 10×128 dimensional matrix, where each behavior has a 128 dimensional feature vector, and includes a total of 10 electroencephalogram responses of visual events, and F2 is a 12×128 dimensional matrix, and includes response features of another 12 visual events. In the above step, the first optimization window W1 (length 15) and the second optimization window W2 (length 21) are generated based on the statistical difference between the stationary signal and the feature signal.
Then for the first eigenvector V11 in F1 (1 x 128 dimensions), its convolution output c11:c11=v11×w1 with W1 can be calculated as in equation (1), where is a convolution operation. Since W1 has a length of 15, C11 is a vector of 1X 114 dimensions. The other 9 eigenvectors in F1 are subjected to similar convolution operation, and all the outputs are spliced according to a formula (2), so that an FE1 matrix which is a matrix with 10 multiplied by 114 dimensions can be obtained.
Similarly, for the first eigenvector V21 of F2 (1×128 dimensions), c21=v21×w2 is calculated, where W2 is 21 in length, and thus C21 is a1×108-dimensional vector. And (3) performing similar processing on all 12 vectors in F2 and splicing according to a formula (4), so as to obtain the FE2 matrix, wherein the dimension is 12 multiplied by 108.
Finally, the FE1 and the FE2 respectively contain the corresponding visual event characteristics under two states of rational thinking and intuitive thinking, and effective characteristic support is provided for subsequent fusion and classification.
Extracting a first stable signal characteristic and a second stable signal characteristic in the steps; and extracting the first visual event characteristics and the second visual event characteristics in the steps. And fusing the two types of features to construct a composite feature sample corresponding to the visual event.
First, since the row vector dimensions of FP1, FP2, FE1, and FE2 may not be exactly the same, it is necessary to dimension them first so that all row vectors have the same length. It is assumed that the row vector dimensions of the four feature matrices are changed to N1, N2, N3, and N4, respectively, after the equal length processing. For ease of representation, the present application recommends FP1 as a (k1×n1) matrix, FP2 as a (k2×n2) matrix, FE1 as a (pxn3) matrix, and FE2 as a (qxn4) matrix.
The key to this is to effectively fuse these four types of features. Since the stationary signal features and the visual event features respectively reflect the brain electrical patterns tested in different cognitive states, combining them together helps to improve the discrimination capability of subsequent classification. The specific fusion strategy needs to weigh the contribution degree of various characteristics to classification and determine a reasonable fusion weight. The application adopts a self-adaptive weighting method based on the sample cohesion degree, so that the weight value can be automatically matched with the internal distribution characteristics of the data.
Firstly, defining a sample cohesion degree scoring function J, which represents the tightness degree between samples in the same category:
j (X) =Σ (i=1 to N) Σ (j=1 to N) exp (- |xi-xj||2/2σ+|2) (1).
Where x= { X1, X2,..xn } is the sample set, each Xi is a D-dimensional feature vector, the expression vector. Is used for the two norms of (2), sigma is the root mean square distance between all vectors in X. It can be seen that J (X) is actually a sum of the similarities between all vectors in X, with a larger value indicating a higher degree of cohesion.
The current goal is to find a set of optimal weights w1, w2, w3, w4 such that the weighted fused feature set f=w1fp1+w2fp2+w3fe1+w4fe2 takes the maximum value on J (F), i.e. has the highest degree of cohesion. Here, multiplication of matrix corresponding elements is represented. Since FP1, FP2, FE1, FE2 may contain different numbers of samples, it is necessary to splice them into equal length sample sequences first and then calculate the cohesive degree J (F) after fusion. Assuming that the length of the spliced sample sequence is N, the objective function can be expressed as max J (F) =max Σ (i=1 to N) Σ (j=1 to N) exp (- |fi-fj|2/2σ+2) w1, w2, w3, w4 (2) s.t. w1+w2+w3+w4=1; w1, w2, w3, w4 > =0.
Where Fi is the i-th sample vector in F. The constraint on this optimization problem is that the sum of weights is 1 and is not negative. The constrained optimization problem may be solved using the lagrangian multiplier method. The specific practice is to construct a Lagrangian function:
l (w 1, w2, w3, w4, λ, μ1, μ2, μ3, μ4) = -J (F) +λ (w1+w2+w3+w4-1) - μ1w1- μ2w2- μ3w3- μ4w4. Wherein λ, μ1, μ2, μ3, μ4 are Lagrangian multipliers. Let the partial derivatives of L for w1, w2, w3, w4 be 0, a system of equations ∂ L/∂ w1=0 ∂ L/∂ w2=0 ∂ L/∂ w3=0 ∂ L/∂ w4=0 w1+w2+w3+w4=1 can be obtained.
Substituting the above formula into the specific form of J (F) and solving the values of w1, w2, w3 and w4 to obtain the optimal weight. With these weights, a fused composite feature set F can be generated as follows: f=w1fp 1 +w2Fp2 +w2 FP 2; f contains K1+K2+P+Q samples, each sample being an N-dimensional feature vector. The method combines stable signal characteristics and visual event characteristics, and reflects the difference of brain electric modes of the tested brain under different cognitive states. It is worth mentioning that the method for solving the optimal weight is an unsupervised strategy, and is completely based on the characteristics of the sample distribution itself without depending on label information, so that the method has a strong generalization capability.
To assist understanding of the above procedure, a specific example is given in which four matrices of FP1, FP2, FE1, FE2 extracted in the above steps are assumed to be FP1 (10×20), FP2 (15×30), FE1 (20×50), FE2 (25×40), respectively. Then they need to be dimension unified first, and after being unified, they become FP1 (10×60), FP2 (15×60), FE1 (20×60), FE2 (25×60), respectively, i.e. all row vectors have a length of 60.
These four matrices are then concatenated to give a sample sequence F of length 10+15+20+25=70, with the first 10 samples coming from FP1, then 15 from FP 2. From formulas (1) (2) (3) (4), the optimal weights are w1=0.2, w2=0.3, w3=0.3, w4=0.2. Substituting the weights into the formula (5) to generate a fused composite feature set F, which is a (70 multiplied by 60) matrix, and fusing stable signal features and visual event features.
And step S107, training a visual event classification model to be trained according to the composite characteristic sample, and detecting the visual event by the trained visual event classification model.
Specifically, a visual event classification model is trained by utilizing the composite characteristic sample, and an electroencephalogram signal classifier with optimized window parameters set for the visual event is established, so that the visual event can be effectively identified.
The training of the visual event classification model to be trained is completed according to the composite characteristic sample, and the visual event classification model comprises the steps of inputting vectors corresponding to the composite characteristic sample into a convolution layer of the deep convolutional neural network model, and outputting predicted visual events and corresponding predicted probability information; acquiring a standard visual event corresponding to the composite characteristic sample; calculating a loss function corresponding to the predicted visual event, the predicted probability information and the standard visual event; and optimizing the weight parameters of the deep convolutional neural network model according to the loss function based on a back propagation algorithm to finish training the visual event classification model.
By adopting a deep Convolutional Neural Network (CNN) as a classification model, the CNN structure can effectively capture time and space characteristic modes in the electroencephalogram signals. First, the composite feature sample F is divided into a training set Ftrain and a test set Ftest, where Ftrain contains N samples { (x 1, y 1), (x 2, y 2), (xN, yN) }. Each xi is a sample vector in F, the dimension is D, yi is the label corresponding to the dimension, and the value is 1 (rational thinking state) or 2 (intuitive thinking state). The input layer of the DCNN model accepts D-dimensional vectors, and the output layer contains two types of visual event labels corresponding to two neurons. The hidden layer is composed of a plurality of convolution layers and a pooling layer.
The first convolution layer carries out one-dimensional convolution operation on the D-dimensional input vector, and extracts a local characteristic mode. Let the layer contain K1 convolution kernels, each kernel length L1, step size S1. Let W1 ε R (K1×L1) be the convolution kernel weight matrix and b1 ε R (K1×1) be the bias term, then the first layer output feature map is x1=σ (W1x+b1) εR (K1× (D-L1)/S1+1). Where one-dimensional convolution operation is represented, σ is the activation function (e.g., reLU, etc.). The first pooling layer performs Max-Pooling operation on X1, the pooling core size is P1, the step length is Q1, and X1' E R (K1X ((D-L1)/S1+1-P1)/Q1+1) is output.
And so on, the output of the first layer is Xl' ∈r++kl× ((D- Σ (i=1 to l-1) (Li-Pi)/pi=1 to l-1si+1-Pl)/ql+1)). Where Kl is the number of convolution kernels of the layer, ll is the kernel length, sl is the step size, pl is the pooling kernel size, and Ql is the pooling step size. Finally, the output Xl ' of the last convolution layer is mapped into two neurons through a full connection layer, and probability values y '1 and y '2 of two types of visual events are corresponding.
Given a training sample (x, y), its loss function is L (y, y ') = -ylog (y ') - (1-y) log (1-y '). Where y ' = [ y '1, y '2] is the output of the CNN model, y is the actual label, y=1 represents the rational thinking state, and y=2 represents the intuitive thinking state. The Loss function for all samples is averaged to get the Loss for the whole model, loss=1/N Σi=1 to NL (yi, y' i). In the training process, a back propagation algorithm is adopted to optimally adjust all weight parameters (including convolution kernel weights, bias items, full-connection layer weights and the like of each layer) of the CNN, so that the Loss is minimized. The optimization method can adopt random gradient descent (SGD), adam and the like. And super parameters (such as learning rate, batch size and the like) are adjusted according to the loss curve, so that overfitting is prevented.
Let the trained CNN model be fCNN (x) and its output be [ y '1, y'2], then for the new test sample x, the method of determining its category is y=argmax (y '1, y' 2). Namely, the visual event category corresponding to x is determined to be the one with larger CNN output probability value. In the training and testing stage, regular terms (such as L1/L2 norms and the like) can be introduced to constrain CNN model parameters, so that generalization capability is further improved. The performance of the model is evaluated on a test set Ftest, and common indexes include classification accuracy acc, accuracy prec, recall recall, F1 score, and the like.
The CNN model described above may be optimized for improved classification accuracy. One approach is to add a temporal convolution layer before the CNN input layer to encode the original sample signal. Let the sample signal length be T, the word embedding dimension be M, the time convolution kernel length be TK, the step length be TS, the time convolution output be Xt E R (M× (T-TK)/TS+1). Wherein the weight matrix Wt epsilon R (M x TK) of the convolution kernel is obtained through training learning. Xt after the time convolution layer coding is used as the input of CNN, and the time dynamic characteristics of the signal can be effectively captured. Another optimization is to add the convolutional layer input to the output using residual connection: xl '=xl' +xl. The residual structure is beneficial to deepening the network depth and acquiring abstract features of higher level.
In addition, if time series tag information exists, a long-short-term memory network (LSTM) layer can be further overlapped on the CNN to capture the time dependent mode of the sample sequence. Let the last layer of feature mapping of CNN be X '∈R++K×T', T 'be the time step number, then LSTM output is h_t=σ (W_ hX' t+U_hh (T-1) +b_h), where σ is the nonlinear activation function, W_h, U_h, b_h are the input weight matrix, recursive weight matrix and bias of LSTM, respectively. Finally, the last hidden state of the LSTM is mapped into two classification probability values through a full connection layer.
Examples of CNN model parameters are a first convolution layer convolution kernel number k1=64, a length l1=5, a step s1=1, a first pooling kernel size p1=4, and a step q1=2. The second convolution layer convolution kernel k2=128, length l2=5, step s2=1, second pooling kernel size p2=4, step q2=2. The time convolution layer word embedding dimension m=32, the kernel length tk=3, and the step length ts=1. The LSTM layer hidden unit number is 128. The output dimension of the full connection layer is 2, and the full connection layer corresponds to two types of visual events. Using the ReLU activation function, the Adam optimizer has a learning rate of 0.001 and a batch size of 32. An L2 norm regularization term is introduced, and the weight decay coefficient is 0.0005. The model was trained on a synthetic dataset containing 10000 samples with a test accuracy of 95.7%.
Through the CNN classification model, the cognitive state of the tested person can be automatically identified from the composite characteristic sample of the brain electrical signal, namely, the rational thinking or the intuitive thinking corresponding to the visual event, the cognitive activity state of the tested person is objectively identified, the instantaneous brain dynamics in task execution is captured, and the cognitive function is more finely depicted and evaluated. For example, in some cases where significant decisions need to be made, it is important to be able to balance the manageability analysis with the intuitive judgment. By electroencephalogram analysis, if the tested brain mainly stays in a rational thinking state, a left brain area is continuously activated, more logic reasoning and meticulous analysis are shown, but some intuitive clues and overall grasp can be ignored, so that decision is on the whole or lost, on the contrary, if the tested brain stays in an intuitive thinking mode for a long time, a right brain area is continuously and highly activated, too subjective and straight white judgment can be generated, the rational analysis is lacked as support, and a decision result can have important omission or defects. Therefore, the current cognitive state of the tested person can be accurately identified through the electroencephalogram analysis technology, unprecedented physiological reference data can be provided for psychological assessment, the assessment result is not only dependent on behavioral measurement or subjective report, but also can directly peep the operation mechanism of the brain, so that all cognitive ability performances of the tested person can be comprehensively and objectively assessed, and more valuable guidance comments are provided for psychological development and cognitive training of the individual person.
Compared with the prior art, the application has at least the following beneficial effects:
1. The method has the advantages that a strategy of stable-salient fusion modeling is provided, autonomous optimization adjustment of feature extraction intervals is realized, the electroencephalogram transformation rules under different cognitive states can be matched, key features under stable and excited states are captured, and a subsequent model based on the strategy can better describe and predict visual events. The self-adaptive system of the strategy has an expanded application value for detecting other types of events.
2. The stable electroencephalogram signal assists in salient feature extraction, the two-state difference is utilized to set the individuation time window parameter, uncertainty of subjective assumption and experience estimation is avoided, and visual event related feature extraction is more stable. Meanwhile, the stability characteristics are used as new sample attributes to be integrated into classification judgment, so that behavior detection and psychological assessment models based on electroencephalogram are enriched.
3. Compared with a direct modeling method for the source electroencephalogram, the modeling is carried out by utilizing the stable-salient difference in stages, the operation efficiency is higher, the requirements on equipment are lower, and the application range is wider. The potential for computational and use cost reduction is greater.
In order to execute the detection method of the psychological assessment visual event based on the electroencephalogram signal corresponding to the embodiment of the method, corresponding functions and technical effects are achieved. Referring to fig. 2, fig. 2 is a block diagram illustrating a detection apparatus 200 for psychological assessment of visual events according to an embodiment of the present application. For convenience of explanation, only a portion related to the present embodiment is shown, and the detection apparatus 200 for psychological assessment visual event provided in the embodiment of the present application includes:
An image acquisition module 201, configured to acquire brain wave images of a user at a plurality of moments, and form an original brain wave time series signal according to a plurality of the brain wave images;
a reference acquisition module 202, configured to acquire first reference feature signals of each two adjacent brain wave images in the original brain wave time sequence signal, acquire second reference feature signals corresponding to each two adjacent first reference feature signals, and acquire third reference feature signals corresponding to each two adjacent second reference feature signals;
A difference calculation module 203, configured to calculate a first difference corresponding to the first reference feature signal and the second reference feature signal and a second difference corresponding to the second reference feature signal and the third reference feature signal at the same time;
A signal extraction module 204, configured to extract a first stable signal set and a second stable signal set from the original brain wave time series signal according to the first difference and the second difference, respectively;
The feature obtaining module 205 is configured to remove the first stationary signal set and the second stationary signal set from the original brain wave time series signal, and obtain a first feature signal set and a second feature signal set;
The signal fusion module 206 is configured to perform feature fusion on the first stationary signal set, the second stationary signal set, the first feature signal set, and the second feature signal set, and obtain a composite feature sample;
And the training completion module 207 is configured to complete training of a visual event classification model to be trained according to the composite feature sample, and the visual event classification model after training is completed detects a visual event.
The above-described detection device 200 for psychological assessment visual event can implement the detection method for psychological assessment visual event based on brain electrical signal according to the above-described method embodiment. The options in the method embodiments described above are also applicable to this embodiment and will not be described in detail here. The rest of the embodiments of the present application may refer to the content of the above method embodiments, and in this embodiment, no further description is given.
Fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 3, the computer device 3 of this embodiment includes: at least one processor 30 (only one is shown in fig. 3), a memory 31 and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, the processor 30 implementing the steps in any of the method embodiments described above when executing the computer program 32.
The computer device 3 may be a smart phone, a tablet computer, a desktop computer, a cloud server, or other computing devices. The computer device may include, but is not limited to, a processor 30, a memory 31. It will be appreciated by those skilled in the art that fig. 3 is merely an example of the computer device 3 and is not meant to be limiting as the computer device 3, and may include more or fewer components than shown, or may combine certain components, or different components, such as may also include input-output devices, network access devices, etc.
The Processor 30 may be a central processing unit (Central Processing Unit, CPU), the Processor 30 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 31 may in some embodiments be an internal storage unit of the computer device 3, such as a hard disk or a memory of the computer device 3. The memory 31 may in other embodiments also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the computer device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the computer device 3. The memory 31 is used for storing an operating system, application programs, boot loader (BootLoader), data, other programs etc., such as program codes of the computer program etc. The memory 31 may also be used for temporarily storing data that has been output or is to be output.
In addition, the embodiment of the present application further provides a computer readable storage medium, where a computer program is stored, where the computer program is executed by a processor to implement the steps in any of the above-mentioned method embodiments.
Embodiments of the present application provide a computer program product which, when run on a computer device, causes the computer device to perform the steps of the method embodiments described above.
In several embodiments provided by the present application, it will be understood that each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present application, and are not to be construed as limiting the scope of the application. It should be noted that any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art without departing from the spirit and principles of the present application are intended to be included in the scope of the present application.