Affective computing is a key technology for human–computer interaction (HCI) in the future era of Internet of thing (IoT) [1], which makes it possible for machines and computers to realize human’s emotion and mentality in real time. Furthermore, it can give appropriate responses and services based-on human’s current mental status. Related databases have been developed for related researches on affective computing. Among the open-source databases available on the Internet, FEEDTUM [2] and Berlin Database of Emotional Speech (EmoDB) [3], released in 2005, used facial video and speech for emotion recognition, respectively. They predicted basic emotion of people, such as anger and happy, by single type of signal.

As intelligent IoT develops, more and more wearable devices are equipped with different kinds of sensors. We can effectively get various signals sensed from subjects and have the access for continuously monitoring. For this scenario, physiological signals can be good inputs for affective computing framework, since we do not express our emotions on faces and make sounds every time. Integrating multiple-channel of physiological signals for emotion recognition and aggregating machine learning classifiers for ensemble learning have gain attention in recent years. DEAP database [4] proposed in 2012 has been experimented in many works [5,6], it contains several types of physiological signals which were recorded when subjects were watching music videos. In DEAP, emotions perceived by subjects are self-assessed on two affect dimensions, valence and arousal. Valence tells how positive or negative the emotion is and arousal indicates the intensity of emotion. In 2016, ACERTAIN database [7] was proposed that first used commercial wearable sensor to collect data and added personality into experiments. AMIGOS database [8] proposed in 2017 contains stable data, several types of videos, and elements for other related research, such as social context and mood. These all indicate that multi-modality analysis of physiological signals is a trend of affective computing. Therefore, how to effectively process different physiological signals and further extract critical features from different channels for emotion classification have been a big challenge.

The aim of this paper is to establish reliable emotion recognition framework based on physiological signals. To improve the accuracy of emotion recognition, we firstly extract the entropy domain features to quantify the regularity and randomness of signal, which has high potential to represent the different levels of emotion. Next, we apply the XGBoost [9] to enhance the performance of classification which has high scalability and efficiency for training and has the ability to learn from high dimension data without using high complexity feature selection algorithms.

The rest of the paper is organized as follows. Section 2 describes common methods for emotion recognition task. Section 3 illustrates the enhanced methods for feature extraction and machine learning engine. Section 4 shows the experimental results and Sect. 5 concludes.

The flow chart of the emotion recognition framework in this study is shown in Fig. 1. Three kinds of physiological signals are inputs and the output is the prediction for high or low of affect dimensions. Three other blocks including pre-processing, feature extraction, and machine learning engine will be introduced as below.

Flow chart of general emotion recognition framework.

Full size image

2.1Pre-processing

We use the processed data provided by the official website of AMIGOS dataset. In AMIGOS database, each subject watched both long and short videos as stimulus for emotion. Because long-length video data contain multiple values for valence and arousal, only short-length video data are used in this work to reduce the uncertainty. There are total 40 subjects and 16 short-length videos in database. Each video of each subject is seen as one data. Each of data has 14 channels for Electroencephalography (EEG), 2 channels for Electrocardiography (ECG), and 1 channel for galvanic skin response (GSR). 7 subjects whose physiological signals of videos contain some missing values are removed (ID number: 9, 12, 21, 22, 23, 24, 33). Therefore, there are totally (40–7) (subjects) × 16 (videos) = 528 data. We further checked the data stability. First Channel of ECG signals suffer from fewer noises compared to second channel and are used for further analyzing. In addition, GSR signals are filtered by the low-pass filter with cut-off frequency 3 Hz to remove abnormal high frequency noises.

In this work, we do binary classification for valence and arousal. Self-assessments of each subject on two affect dimensions provided in AMIGOS database is used for labels. They are originally values between [1,9] and needed to be transformed to either positive class or negative class label, which represents high and low, respectively. Figure 2 shows two different ways to process. Directly cut by threshold 5 is intuitive, but hard to distinguish one’s relative high and low emotion. In this way, values may highly accord to personal tendency to rate high or low scores, which results in imbalance of labels and decrease in performances. Our method is to define labels by subject-dependent mean value among all 16 videos of each person. Using mean value instead of median value can avoid ambiguity whether the median is labeled positive class or negative class.

(a) Define labels by comparing with threshold score 5. (b) Define labels by comparing with subject-dependent mean among all videos.

Full size image

2.2Feature Extraction

Feature extraction is implemented on all three kinds of signals with different ways to reduce the input dimension before using machine learning methods. Features illustrated in this Section are slightly different from the feature set in AMIGOS database. Features extracted among physiological signals are shown in Table 1.

GSR Signal:

Partly follow methods in [8,10], skin response (SR) related features (18) are first extracted and skin conductance (SC) signal is got by computing SR signal’s reciprocal. SC signal is then normalized and used to extract features (4). Next, skin conductance slow response (SCSR) and skin conductance very slow response (SCVSR) is got by low-pass filtering normalized SC signal at 0.2 and 0.08 Hz, respectively. Afterwards, some SCSR related features are extracted (4). Last, after de-trending SCSR and SCVSR by using empirical mode decomposition (EMD) methods [11] to remove the last half intrinsic mode functions (IMFs), some related features are extracted.

ECG Signal:

Partly follow methods in [8,10], 0–6 Hz spectral power features are first extracted (60, per 0.1). Then, R-R interval (RRI) series is calculated after detecting R peaks in ECG. Heart rate variability (HRV) and heart rare (HR) time series can be computed by using RRI. Finally, HRV related features (11) and HR related features (6) are extracted.

EEG Signal:

Follow the methods in [4], average power spectral density (PSD) of theta band (4–7 Hz), slow alpha band (8–10 Hz), alpha band (8–13 Hz), beta band (14–30 Hz), gamma band (31–47 Hz) of each EEG channel are extracted (5 × 14 = 70). Also, asymmetry of PSD of 5 bands between 7 pairs of EEG channel are extracted (5 × 7 = 35).

2.3Machine Learning Engine

This block consists of a feature selection method and a machine learning classifier, as shown in Fig. 3a. Feature space which contains 214 features is comparatively big while we only have 528 (16 subjects * 33 videos) data in AMIGOS database. Thus, feature selection is applied to eliminate redundant features and reduce the model complexity of machine learning algorithm in next stage. This is helpful for enhancing overall performance and sparing lots of computation resources.

(a) Overview of machine learning engine block. (b) Machine learning engine block in AMIGOS paper. (c) Machine learning engine block of simple benchmark.

Full size image

In Fig. 3b, overview of machine engine block in AMIGOS database is shown. For feature selection method in original paper of Amigos database [8], Fisher’s linear discriminant (FLD)J [12] is calculated over each feature, which is defined as

Afterward, user can decide how much discriminant features they would like to select based on validation sets and pick up features with the highestJ values. For machine learning classifier, Gaussian Naïve Bayes (GaussianNB) classifier is used. Assuming features are independent and Gaussian distributed, GaussianNB is given as

and calculate the probability of each sample for each class, whereF is feature set andC is the classes set. These two methods are not powerful and lead to relatively low f1-score shown in [8].

In this work, we first implement support vector machine (SVM) [13,14] based approach as the benchmark, as shown in Fig. 3c. SVM is a famous tool in the field of machine leaning and it has good performance on classification and prediction over many applications. Compared with other classifier, SVM can deal with small-size datasets well, since it uses only support vectors to construct hyperplane. When it comes to feature selection, recursive feature elimination (RFE) algorithm [15] can efficiently remove irrelevant features and be suit for SVM. RFE-SVM eliminates features with smallest weight in SVM model on a sequential backward selection (SBS) based process. The progress continues until one feature is left and the features combined with SVM model with the highest performance is outputted.

3.1Entropy Domain Features

Non-linear entropy domain features, such as sample entropy, permutation entropy based features, are widely used on physiological signals. Extracted entropy value can help quantify the regularity of signal and thus be applied on medical diagnosis. For this emotion recognition work, three types of entropy domain features, including refined composite multiscale entropy (RCMSE), turning point ratio (TRP), and Shannon entropy, are applied to measure the complexity of physiological signals. The details are shown as follows.

Refine Composite Multiscale Entropy (RCMSE) [16]:

Multiscale entropy (MSE) [17] has been used widely to evaluate physiological control mechanisms, such as atrial fibrillation and Alzheimer’s disease [18]. RCMSE is an improved version of MSE. It reduces the possibility of undefined value problem when the signal length is short and has a better accuracy for entropy estimation. The concept of RCMSE is using different scales of local matching pattern to compute the regularity of the signal. There are two steps of RCMSE. First step: time series of signal is coarse-grained into multiscale series. For each scale factor\( \tau \),\( \tau \) series are generated by average\( \tau \) points in non-overlapping windows and each of series overlap\( \tau \)−1 points with neighbor series. Thej-th point ofk-th coarse-grained series,\( \varvec{y}_{k}^{{\mathcal{(}\tau \mathcal{)}}} = \left\{ {y_{k,1}^{{\mathcal{(}\tau \mathcal{)}}} y_{k,2}^{{\mathcal{(}\tau \mathcal{)}}} y_{k,3}^{{\mathcal{(}\tau \mathcal{)}}} \ldots y_{k,p}^{{\mathcal{(}\tau \mathcal{)}}} } \right\} \), with scale factor\( \tau \) of signalx is defined:

In the conventional MSE algorithm, output of coarse graining is only first series of each scale factor. Second step: averaged sample entropy of each\( \tau \) is calculated as below:

wherem is the matching pattern length andr is the similarity criterion. In (4),\( n_{k,\tau }^{m} \) is the number of two sets of simultaneous data points of lengthm inkth series have the difference < r. When the ratio of\( \sum\nolimits_{k = 1}^{\tau } {n_{k,\tau }^{m + 1} } \) and\( \sum\nolimits_{k = 1}^{\tau } {n_{k,\tau }^{m} } \) is small, sample entropy would be large, which means high complexity of signal. Value of RCMSE is undefined only when\( \sum\nolimits_{k = 1}^{\tau } {n_{k,\tau }^{m + 1} } \) = 0, which means\( n_{k,\tau }^{m} \) ofk series are all zero. By summation of matching pattern of multiple series, RCMSE has less probability to be undefined than conventional MSE.

In this work, RCMSE is applied to RRI series of ECG signals. Because the length of ECG signals are relatively short (each video 55 s–155 s), the scale factor\( \tau \) s set up to be 3. In addition, we set the matching patternm up to be three and the similarity criterion r is set to be 0.2 of standard deviation.

Turning Points Ratio (TPR):

TPR is proposed on the basis of nonparametric “Runs Test” to evaluate the randomness in a time-series [19] and the idea was used in RRI of ECG signal [20]. The concept of TPR is to measure the complexity of the signal by number of turning points compared to total points. Turning point is found by comparing each point with left and right neighbor points and TPR is calculated as follows:

whereN is the length of the signalx. Besides original TPR, we extracted modified TPR (MTPR) on signals. The procedure of MTPR is to use EMD methods to extract trend of the signal first, in order to remove trivial peaks of the signal. Next, compute TPR as (3) on the extracted trend.

In this work, TPR and MTPR are calculated on RRI of ECG signals and GSR signals. EEG signal has less information on time series and is skipped.

Shannon Entropy:

Shannon entropy is commonly calculated in the domain of information theory. It is defined for a given discrete probability distribution, using probability of each symbol to measure the uncertainty or randomness of the data. However, almost every point has different values. We can’t see each of different points as a new symbol, otherwise almost the same value of Shannon entropy we would get from different signals. In other way, we classify every points into one group of the group set. First, outliers that have larger differences than three standard deviations with mean are removed. Second, we sort rest data points into N groups, which are equally divided between max and min value. Last,\( p_{i} \) of eachith group is calculated and Shannon entropy we can get by:

The concept of Shannon entropy is to observe the complexity of the signal by overall distribution. Each group is seen as a symbol.

In this work, Shannon entropy is calculated on RRI of ECG signals and GSR signals. Since the optimal group number is data dependent, we apply total 4, 8, 16, 32, 64 groups for simulation.

For these three enhanced feature extraction methods, total 26 features are added into original feature space. The details are listed in Table 2.

3.2Extreme Gradient Boosting (XGBoost)

In this part, we would like to change the content of the machine learning blocks mentioned before in Sect. 2.3, where RFE based SVM approach was used. Among all the machine learning algorithms, gradient boosting tree based model [21] has shown in many applications in different domains. XGBoost [9] is an efficient and scalable gradient boosting machine, which has won lots of machine competitions in recent years [22,23]. It is an ensemble model consisting of sets of classification and regression tree (CART). While XGB is used for supervised learning problems and we use training data\( x_{i} \) to predict a target variable\( y_{i} \), the model can be described in the form:

whereK is the total number of trees,f_k forkth tree is a function in the functional spaceF, andF is the set of all possible CARTs. In the training, each of new-trained CART will try to complement the so-far residual. Objective function been optimized at (t + 1)th CART is described:

Wherel() denotes the training loss function,\( y_{i} \) the is ground truth, and\( \hat{y}_{i}^{{\mathcal{(}t\mathcal{)}}} \) is the prediction value at stept.\( \Omega \left( {} \right) \) given by

is the regularization term, whereT are the number of leaves and\( w_{j} \) is the score onjth leaf. When (9) is optimized, Taylor’s expansion is used so that gradient descent can be used for different loss functions. Furthermore, feature selection is no need when we use XGBoost approach. During training period of XGBoost, good features would be chosen as node in trees, which means features not used are abandoned.

In this work, we use the scikit-learn API for XGBoost classification. The inputs of XGBoost are total 240 features (214 traditional features + 26 entropy domain feature) and the outputs are prediction results for valence or arousal. For loss function in (8), logistic loss function is set. The details of used features and performances are shown in next Section.

Single trial classification for two affect dimensions, arousal and valence, is experimented and the flow is shown in Fig. 1. After signal is pre-processed and transformed into features, all features are normalized to [1]. Leave-one-subject-one approach is conducted to evaluate performance. That is, every time one of the 33 subjects is leaved as test set and machine learning engine is trained using remaining 32 subjects’ features. When the procedure is repeated 33 times, final performance is calculated by averaging 33 values in each procedure.

There are 3 processing schemes that are compared.

• Scheme_1: Using the feature set and the method described in original AMIGOS database [8], where FLD and GaussianNB were used for machine learning engine block.

• Scheme_2: Using the feature set illustrated in Sect. 2.2 and RFE-SVM for machine learning engine block.

• Scheme_3: Using new feature set with entropy domain features and XGBoost for machine learning engine block.

Table 3 shows the comparison of f1-score and accuracy over three schemes, F1-score is the harmonic mean of precision and recall and here we average f1-scores of positive and negative class as final value. Four scenarios were experimented: using only GSR, ECG, EEG, and using all modalities. Table 4 shows the features used in trees of XGBoost classifier for classification of valence and arousal.

F1-score for Scheme_1 is directly obtained from [8]. For dimension valence and arousal, best accuracy is about 80% and 68%, respectively. Scheme_3 outperforms Scheme_1 and Scheme_2 in almost every scenario on both affect dimensions, especially in valence dimension, outperforms by more than 10% of accuracy. Using all-modalities for prediction can have highest performances except for f1-score in arousal. In our experiments (Scheme_2 and Scheme_3), using GSR channel to predict can have the best performance for both valence and arousal dimensions, when only one channel is used. However, the features extracted in GSR channel is the least among three modalities, which indicates that the quality but not the quantity of features lead to better prediction ability.

In this paper, we proposed the scheme including both entropy domain features and XGBoost. We enhance feature extraction methods which are entropy domain and helpful for evaluating the complexity of physiological signals. On the other hand, XGBoost classifier which gains popularity in recent years is used for learning and prediction. The proposed scheme can reach the performance of approximately 80% and 68% accuracy on valence and arousal dimension, respectively. It outperforms the processing scheme in original paper of AMIGOS database and the scheme that contains common features and traditional SVM model.