1. Introduction

The emotional brain is a network of key brain areas including the prefrontal cortex (PFC), amygdala, hypothalamus and anterior cingulate cortex (ACC) (Dalgleish, 2004; Tovotet al., 2015). A small part of the sensory information entering the brain goes to the amygdala. The amygdala can respond quickly to emotionally relevant stimuli and for instance prepare the body for action without much stimulus processing. After the amygdala, processing continues through the cingulate cortex, the ventromedial prefrontal cortex and finally the dorsolateral prefrontal cortex. Only in the dorsolateral prefrontal cortex is the processing stream through the amygdala integrated with the more cognitive processing stream from the sensory cortices. Emotional experience and affective appraisal are the results of the integration of both processing routes taking into account the context and previous experiences (van Erpet al., 2016). This integration and interpretation of information is a typical function of the prefrontal cortex (Isotaniet al., 2002). This network, starting with the amygdala, receives information from all sensory systems (LeDoux, 2007; McDonald, 1998), and it is therefore not surprising that emotional information perceived via different sensory systems interacts with each other (Schreuderet al., 2016). For instance, the sensation of wine depends upon the background music (see Spence, 2015, for a review), and the emotional experience of music depends on whether one can view the musician (Vineset al., 2006). In general, multisensory interactions are typically observed when the information from different sensory modalities is temporally or spatially aligned (Alais & Burr, 2004; McGurk & MacDonald, 1976; Morein-Zamiret al., 2003; Philippiet al., 2008; Shamset al., 2000; Shipley, 1964; Van der Burget al., 2011, 2013a; Vroomen & De Gelder, 2000), and decreases with increasing asynchrony (Noelet al., 2015; Slutsky & Recanzone, 2001; Van der Burget al., 2010, 2014; Van Wassenhoveet al., 2007).

However, what we perceive via our different sensory modalities is not solely determined by what we sense at a given moment in time. Indeed, numerous studies have shown that our percept is also biased by information we processed in our immediate past (Alaiset al., 2017; Fischer & Whitney, 2014; Fritscheet al., 2017; Harveyet al., 2014; Kiyonagaet al., 2017; Libermanet al., 2014; Taubertet al., 2016; Van der Burg & Goodbourn, 2015; Xiaet al., 2016). Such serial dependencies have been observed across different modalities and tasks (e.g., Fornaciai & Park, 2019; Lau & Maus, 2019; Libermanet al., 2018; Manassiet al., 2018; Togoliet al., 2021)). There also appears to be a serial effect for the affective appraisal of emotional stimuli. For instance, in our recent study, participants viewed a random sequence of different food images and reported their affective appraisal of each image in terms of valence and arousal (Van der Burget al., submitted). For both measures, we conducted an intertrial analysis, based on whether the rating on the preceding trial was low or high. The analysis showed that valence and arousal ratings for a given food image were both assimilated towards the ratings on the previous trial (i.e., a positive serial dependence). A positive serial dependence was also observed when participants rated the emotional expression of faces (Libermanet al., 2018) and when they made aesthetic judgements about artwork (Kimet al., 2019).

A feasible explanation for this positive serial dependence may be a residual activation in the emotional network caused by previous emotional experiences. In other words, an emotional response does not disappear abruptly when a stimulus disappears, but instead slowly decays over time (see Lapateet al., 2017, for an example using face stimuli). As a result, the emotional response to a subsequent stimulus is not a ‘pure’ response to the stimulus itself, but rather a mixture of the residual emotional activation in combination with the activation caused by the current stimulus. Such an averaging process over time also explains why the emotional response to a new image is shifted towards the preceding emotional response (i.e., a positive serial dependence). What is interesting is that serial dependencies are not specific to the visual domain. Indeed, similar short-term dependencies have been reported within the auditory domain when participants were instructed to judge the direction of an auditory frequency sweep (Alaiset al., 2015), the loudness (Holland & Lockhead, 1968; Jesteadtet al. 1977), or the timbre (Piazzaet al., 2018) of a sound. Serial dependencies have also been observed within the tactile domain when rats had to judge each vibrissae (or whiskers) vibration (Hachenet al., 2020).

An intriguing question is whether the affective appraisal of emotional stimuli perceived in one sensory modality also depends on what was processed in the immediate past by different sensory modalities. For instance, does the emotional experience for a given visual or auditory stimulus depend on whether you recently had a pleasant (like viewing a strawberry, or hearing the ocean), or an unpleasant (like viewing a shooting soldier, or hearing a car crash) emotional stimulus. Previous studies found no serial dependence at the perceptual level across the auditory and visual sensory modalities (Fornaciai & Park, 2019; Lau & Maus, 2019). However, since there is a significant overlap between the brain regions that mediate the core dimensions of affect across these different sensory modalities (Satputeet al., 2015), it is likely that there may also be a crossmodal transfer of serial dependency. The goal of this study is to investigate whether serial dependencies for the affective appraisal of emotional stimuli transfer across the auditory and visual modalities. In Experiment 1, we investigate whether the emotional experience transfers from one trial to another regardless of the stimulus modality on successive trials. Given the amodal nature of the amygdala (LeDoux, 2007; McDonald, 1998) and the emotional network, we expect to find a positive serial dependence (consistent with Van der Burget al., submitted) for emotion perception regardless of the stimulus modality on the previous trial ($\mathit{t}-1$) and the current trial (t). Another intriguing, yet unresolved question is whether such a positive serial dependence for emotional stimuli also occurs when the task does not require activation of the higher-order brain areas in the emotional network such as the PFC. Involvement of the PFC is a prerequisite in for instance formulating the affective appraisal of a stimulus but not for merely observing a stimulus. Activation of the complete emotional network may result in a larger residual activity in the network and thus a larger sequential dependence than a partial activation of the network, e.g. the amygdala only. For instance, Tamietto and de Gelder (2010) found evidence that the amygdala exerts some of its functions even when participants were not aware of the content or even presence of an emotional stimulus (see Dianoet al., 2017, for an interesting review regarding the amygdala’s response to emotional stimuli without awareness). In Experiment 2, we investigate whether the emotional experience on the previous trial affects the emotional experience on the current trial even though no emotional judgement is required on the previous trial. If passively observing an emotional stimulus leads to partial activation of the emotional network only, we expect to find a reduced positive serial dependence for emotional stimuli compared to Experiment 1.

2. Experiment 1

The aim of Experiment 1 is to examine whether affective appraisal of visual and auditory stimuli depends on the valence and arousal rating on the preceding trial regardless of the stimulus modality on the preceding trial. In the present study, participants were presented a random sequence of 100 sounds and 100 images and instructed to rate the valence and arousal of each stimulus using the EmojiGrid (see Fig. 1A). Some example images are given in Fig. 1B, and two example trials are depicted in Fig. 1C.

View Full Size

(A) EmojiGrid response tool used to rate the auditory and visual stimuli on each trial (Toetet al., 2018). The horizontal axis represents the valence dimension and the vertical axis represents the arousal dimension (0–100 point scale). (B) Example images used in the present study. (C) Two consecutive example trials for Experiment 1. Participants either saw an image or heard a sound, and were instructed to rate each stimulus using the EmojiGrid.

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

If an emotional response to a given stimulus results in a residual activation when the stimulus disappears, so that the subsequent emotional response becomes a mixture of this residual response and the emotional response triggered by the new stimulus, then we expect the rating for a given stimulus to be higher when the residual activation (caused by the previous trial) is high than when it is low (i.e., a positive serial dependence). Moreover, we expect this positive serial dependence to be independent of the stimulus modality on successive trials, if the emotional response is processed by brain areas that receive input from multiple senses, like the amygdala (LeDoux, 2007; McDonald, 1998).

2.2. Results

Figure 2 illustrates the distribution of the responses in the EmojiGrid for all stimuli used. Here, the mean arousal rating is shown as a function of the mean valence rating for both auditory and visual stimuli. Consistent with previous studies, the distribution of the responses in the EmojiGrid follows a U-shaped pattern (see, e.g., Kanekoet al., 2018; Van der Burget al., submitted), although this pattern was more pronounced for the auditory stimuli than for the visual stimuli.

View Full Size

Distribution of the responses in the EmojiGrid. Mean arousal ratings and mean valence ratings for all stimuli used. Green dots represent the auditory stimuli, while the purple dots represent the visual stimuli. The continuous lines represent the best quadratic fits to the data points for both modalities used (see also Kanekoet al., 2018; Van der Burget al., submitted).

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

For both the valence and arousal ratings for each participant, we first calculated the median rating over all auditory and over all visual stimuli (i.e., the neutral rating for each modality). Subsequently, each stimulus was labelled either ‘low’ or ‘high’ on valence or arousal, if the rating for that particular stimulus was either smaller or equal to or larger than the participants’ median rating, respectively. Then, for all participants, we binned the data into two bins. One bin contained those trials in which the preceding stimulus was labelled ‘low’, and the other bin represented those trials in which the preceding trial was labelled ‘high’. For each participant, we then calculated the mean rating for each bin. The difference between the means of the ‘high’ and ‘low’ bins then represents the serial dependence. Figure 3 illustrates the mean valence (upper panels) and arousal (lower panels) ratings as a function of the stimulus modality on the current (t) and previous trial ($\mathit{t}-1$). The first trial was excluded from further analyses since we were interested in history effects.

View Full Size

Results Experiment 1. (A) Mean valence rating as a function of the stimulus modality on the previous trial ($\mathit{t}-1$) and the valence rating (lowversus high) on the preceding trial for both auditory (left panel) and visual (right panel) stimuli on a given trialt. (B) Mean valence intertrial effect as a function of the previous ($\mathit{t}-1$) modality for both auditory and visual stimuli on a given trialt. The error bars represent the standard error of the mean (SEM).

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

2.2.1. Valence Rating

We conducted a repeated-measures ANOVA on the mean valence rating with stimulus modality on trialt, stimulus modality on trial$\mathit{t}-1$ and valence rating on the previous trial as within-subject variables. Alpha was set to 0.05. The ANOVA yielded a significant effect of stimulus modality on trialt,${\mathit{F}}_{1\mathrm{,}96}=4.33$,$\mathit{p}=0.040$, indicating that the overall mean valence rating was higher for visual (47.9) than for auditory stimuli (46.2; see Fig. 3a). Importantly, the main effect of valence rating on the previous trial was significant,${\mathit{F}}_{1\mathrm{,}96}=21.52$,$\mathit{p}\mathrm{<}0.00005$, as the mean valence rating was significantly higher when the valence rating on the previous trial was high (48.4) than when it was low (45.7). This positive serial dependence neither interacted with the stimulus modality on the current trial,${\mathit{F}}_{1\mathrm{,}96}=2.32$,$\mathit{p}=0.131$, nor with the stimulus modality on the previous trial,${\mathit{F}}_{1\mathrm{,}96}=0.78$,$\mathit{p}=0.380$. Figure 2b illustrates the positive serial dependencies for each modality combination. The three-way interaction also failed to reach significance,${\mathit{F}}_{1\mathrm{,}96}=1.60$,$\mathit{p}=0.210$. All other effects were not significant (allF values < 1).

2.2.2. Arousal Rating

We conducted a second ANOVA on the mean arousal rating with stimulus modality on trialt, stimulus modality on trial$\mathit{t}-1$ and arousal rating on the previous trial as within-subject variables. The ANOVA yielded a significant effect of stimulus modality on trialt,${\mathit{F}}_{1\mathrm{,}96}=17.58$,$\mathit{p}\mathrm{<}0.0001$, indicating that the overall mean arousal rating was lower for visual (41.0) than for auditory stimuli (45.5). The main effect of previous modality failed to reach significance,$\mathit{F}\mathrm{<}1$. Importantly, the main effect of arousal rating on the previous trial was significant,${\mathit{F}}_{1\mathrm{,}96}=63.46$,$\mathit{p}\mathrm{<}{10}^{-11}$, indicating that the mean arousal rating was significantly higher when the arousal rating on the previous trial was high (45.6) than when it was low (40.8). This positive serial dependence interacted neither with the previous modality,${\mathit{F}}_{1\mathrm{,}96}=1.99$,$\mathit{p}=0.161$, nor with the current modality,${\mathit{F}}_{1\mathrm{,}96}=0.66$,$\mathit{p}=0.420$. However, the three-way interaction was significant,${\mathit{F}}_{1\mathrm{,}96}=10.11$,$\mathit{p}=0.002$, suggesting that the serial dependence varied across the different conditions. As is clear from Fig. 3b, the serial dependence was significantly smaller when the stimulus modality on trialt was visual and$\mathit{t}-1$ was auditory compared to when the stimulus modality on trialt and$\mathit{t}-1$ were both auditory,${\mathit{t}}_{96}=2.52$,$\mathit{p}=0.014$, and compared to when the stimulus modality on trialt and trial$\mathit{t}-1$ were both visual,${\mathit{t}}_{96}=2.91$,$\mathit{p}=0.004$. In contrast, the serial dependence was not significantly different when the stimulus modality on trialt was visual and$\mathit{t}-1$ was auditory compared to when the stimulus modality on trialt was auditory and trial$\mathit{t}-1$ was visual,${\mathit{t}}_{96}=1.48$,$\mathit{p}=0.142$. Importantly, the positive serial dependence was significantly different from zero for all four possible combinations (all${\mathit{t}}_{96}$ values ⩾ 2.84, allp values ⩽ 0.006).

We conducted a post-hoc analysis to examine whether the valence and arousal ratings on a given trialt depended on the valence and arousal ratings two trials back (trial$\mathit{t}-2$), respectively (see, e.g., Taubertet al., 2016; Van der Burg & Goodbourn, 2015; Van der Burget al., 2015, submitted)). Note that we excluded the first two trials from further analyses. The valence rating was higher when the valence rating on trial$\mathit{t}-2$ was high (47.2) than when it was low (46.9), but this effect failed to reach significance ($\mathit{p}=0.497$). The arousal rating was higher when the arousal rating on trial$\mathit{t}-2$ was high (43.7) than when it was low (42.7), but this effect failed to reach significance ($\mathit{p}=0.06$).

Taken together, for both the valence and arousal ratings we observed a positive serial dependence, regardless of the modality on the previous trial and the modality on the current trial. Thus, it appears that the emotional content from one trial transfers to the subsequent trial, and this occurs both within and between the different sensory modalities. In Experiment 2, we examine whether this transfer occurs automatically (by passive viewing or listening) or whether an explicit judgement of the stimuli is a prerequisite for this positive serial dependence as it was part of the methods of Experiment 1.

3. Experiment 2

The aim of Experiment 2 is to examine whether passively perceiving an emotional stimulus is sufficient to induce an emotional serial dependence as observed in Experiment 1. Experiment 2 was largely identical to Experiment 1. However, participants were instructed to rate the stimuli only on ‘active’ trials and not to respond on ‘passive’ trials (see also Van der Burget al., 2013b). The participants were informed that the active and passive trials were presented in alternating order, so that they were able to optimally prepare for the task. Furthermore, on each trial, participants were informed about the nature of a trial (i.e., active or passive trial) by presenting either a green or red fixation cross prior the emotional stimulus. In the case of an active trial, participants rated the stimulus immediately after its presentation using the EmojiGrid (like in Experiment 1). In the case of a passive trial, a blank screen was shown for 1000 ms after the stimulus presentation. Two consecutive example trials are illustrated in Fig. 4.

View Full Size

Two consecutive example trials for Experiment 2. Participants either saw an image or heard a sound, and were only asked to rate stimuli on active trials (indicated by a green fixation cross preceding the stimulus presentation), and not on passive trials (indicated by a red fixation cross). Active and passive trials were presented in alternating order, and the stimuli were randomly selected from the entire stimulus set.

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

In Experiment 2, only active trials were analysed, since these are the only trials for which we have valence and arousal ratings. If an explicit emotional judgement is required to activate the full emotional network and therewith increase the residual activity, we expect a reduced serial dependence, as the participants never responded on the previous trial (i.e., a passive trial). In contrast, if passively observing an emotional stimulus also triggers the full emotional network, then we expect a positive serial dependence of the same magnitude after a passive trial. It is important to note that the same participants participated in both Experiment 1 and in Experiment 2, that the order of performing the experiments was counterbalanced, and that the auditory and visual stimuli were identical in both experiments. Hence, in Experiment 2, we used the valence and arousal ratings for each participant and for each specific stimulus in a passive trial as provided in Experiment 1 (see also Van der Burget al., 2019, for a similar appoach).

3.2. Results

We first analysed whether the ratings from the participants were consistent across the two experiments. In Fig. 5, we plot the valence and arousal ratings for both modalities in Experiment 1 as a function of the ratings in Experiment 2 (active trials) for each stimulus.

View Full Size

(A) Mean valence rating in Experiment 1 as a function of the mean valence rating in Experiment 2 (active trials) for each auditory (left panel) and visual stimulus (right panel). (B) Mean arousal rating in Experiment 1 as a function of the mean arousal rating in Experiment 2 (active trials) for each auditory (left panel) and visual stimulus (right panel).

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

We expect a strong positive correlation (approaching 1), if participants respond to the stimuli in a consistent manner across the two experiments. It is clear from Fig. 5 that the ratings between Experiment 1 and Experiment 2 correlated very well (auditory valence: Pearson’s$\mathit{r}=0.98$; visual valence: Pearson’s$\mathit{r}=0.99$; auditory arousal: Pearson’s$\mathit{r}=0.91$; visual arousal: Pearson’s$\mathit{r}=0.95$; allp values < 0.001). That we observe a correlation close to 1 means that we can use the ratings from Experiment 1 as valence and arousal of stimuli presented in passive trials in Experiment 2.

In Experiment 2, only active trials were analysed. Like in Experiment 1, we examined whether the valence and arousal ratings on a given trialt depend on the corresponding ratings on the preceding (passive) trial. As mentioned before, the valence and arousal ratings were based on the responses in Experiment 1 (see Van der Burget al., 2019, for a similar methodology). Furthermore, for both the valence and arousal ratings for each participant, we first calculated the median ratings over all the auditory and all the visual stimuli (i.e., a neutral rating for each modality), but only over the passive trials (as these are the only trials we want to label ‘low’ or ‘high’). Subsequently, each passive stimulus was either labelled ‘low’ or ‘high’ valence or arousal rating, if the rating for that particular stimulus was either less than or equal to or larger than the participants’ median rating, respectively. Then, for all participants, we binned the data (active trials) into two separate bins for each modality. One bin contained those trials in which the preceding passive trial was labelled ‘low’, and the other bin represented those trials in which the preceding passive trial was labelled ‘high’. For each participant, we then calculated the mean rating for each bin. The difference between the means of the ‘high’ and ‘low’ bins then represents the serial dependence. Figure 6 illustrates the mean valence (upper panels) and arousal (lower panels) ratings as a function of the rating on the previous trial, the stimulus modality on the current (t) and previous trial ($\mathit{t}-1$). Again, the first trial was excluded from further analyses since we were interested in history effects.

View Full Size

Results Experiment 2. (A) Mean valence rating as a function of the stimulus modality on the previous trial ($\mathit{t}-1$) and the valence rating (lowversus high) on the preceding trial for both auditory (left panel) and visual (right panel) stimuli on a given trialt. (B) Mean valence intertrial effect as a function of the previous ($\mathit{t}-1$) modality for both auditory and visual stimuli on a given trialt. The error bars represent the standard error of the mean (SEM).

Citation: Multisensory Research 35, 2 (2022) ;10.1163/22134808-bja10064

• Download Figure
• Download figure as PowerPoint slide

3.2.1. Valence Rating

We conducted a repeated-measures ANOVA on the mean valence rating with stimulus modality on trialt, stimulus modality on trial$\mathit{t}-1$ and valence rating on the previous trial as within-subject variables. The ANOVA yielded a significant effect of stimulus modality on trialt,${\mathit{F}}_{1\mathrm{,}96}=6.24$,$\mathit{p}=0.014$, indicating that the overall mean valence rating was lower for visual (44.2) than for auditory stimuli (46.0; see Fig. 6a). The main effect of previous modality was not significant,$\mathit{F}\mathrm{<}1$. The modality × previous modality interaction was marginally significant,${\mathit{F}}_{1\mathrm{,}96}=4.15$,$\mathit{p}=0.044$, and not further analysed. Importantly, the main effect of valence rating on the previous trial was significant,${\mathit{F}}_{1\mathrm{,}96}=48.36$,$\mathit{p}\mathrm{<}0.00001$, as the mean valence rating was significantly higher when the valence rating on the previous trial was high (48.3) than when it was low (41.9). This positive serial dependence did not depend on the modality of the current active trial, as the modality × valence rating on the previous trial interaction was not significant,$\mathit{F}\mathrm{<}1$. However, the interaction between the modality on the previous trial and the valence rating on the previous trial was significant,${\mathit{F}}_{1\mathrm{,}96}=16.01$,$\mathit{p}\mathrm{<}0.001$, indicating that the serial dependence depended on the modality of the previous trial. The serial dependence was larger when the emotional stimulus on the previous trial was visual (9.0) than when it was auditory (3.8). Importantly, these positive serial dependencies were significantly different from zero for both sensory modalities on the preceding trial (auditory:${\mathit{t}}_{96}=4.78$,$\mathit{p}\mathrm{<}0.001$, and visual:${\mathit{t}}_{96}=6.54$,$\mathit{p}\mathrm{<}0.001$). The three-way interaction failed to reach significance,${\mathit{F}}_{1\mathrm{,}96}=1.48$,$\mathit{p}=0.227$.

3.2.2. Arousal Rating

We conducted a final repeated-measures ANOVA on the mean arousal rating (on active trials) with stimulus modality on trialt, stimulus modality on trial$\mathit{t}-1$ and arousal rating on the previous (passive) trial as within-subject variables. The ANOVA yielded a significant effect of stimulus modality on trialt,${\mathit{F}}_{1\mathrm{,}96}=34.35$,$\mathit{p}\mathrm{<}0.001$, indicating that the overall mean arousal rating was lower for visual (41.0) than for auditory stimuli (45.5). The main effect of previous modality was not significant,$\mathit{F}\mathrm{<}1$. Importantly, the main effect of previous arousal rating was significant,${\mathit{F}}_{1\mathrm{,}96}=24.15$,$\mathit{p}\mathrm{<}{10}^{-11}$, as the mean arousal rating was higher when the arousal rating on the previous passive trial was high (43.5) than when the arousal rating on the previous trial was low (40.5). This positive serial dependence did not interact with the modality on the current active trial,$\mathit{F}\mathrm{<}1$. However, like for the valence rating, the positive serial dependence did interact with the previous modality,${\mathit{F}}_{1\mathrm{,}96}=8.06$,$\mathit{p}=0.006$. The serial dependence was larger when the emotional stimulus on the previous trial was visual (4.7) than when it was auditory (1.4). The positive serial dependence was significantly different from zero when the emotional stimulus on the previous trial was visual,${\mathit{t}}_{96}=5.14$,$\mathit{p}\mathrm{<}0.001$, but not when the emotional stimulus was auditory,${\mathit{t}}_{96}=1.74$,$\mathit{p}=0.086$. The three-way interaction was not significant,${\mathit{F}}_{1\mathrm{,}96}=3.16$,$\mathit{p}=0.079$. All other effects failed to reach significance (allF values ⩽ 1.76, allp values ⩾ 0.188).

In Experiment 2, we observed a positive serial dependence for valence ratings, regardless of the stimulus modality on the current and the previous trial. We observed a similar positive serial dependence for the arousal ratings. However, this serial dependence was only significant when the preceding passive trial presented a visual stimulus, and was only marginally significant when the preceding passive trial presented an auditory stimulus. The presence of a serial dependence effect in Experiment 2 is noteworthy as no explicit response was provided on the preceding trial. In fact, since participants were informed that the trial type was presented in alternating order, and since they received an explicit cue (the colour of the fixation cross) about the upcoming trial type, they could have been expected to process the active emotional information in the passive trials less thoroughly.

3.2.3. Between Experiment Analyses

Finally, we examined whether the serial dependencies for both valence and arousal differed between the two experiments. If the task in Experiment 2 (passively observing) only leads to a partial activation of the emotional network, the size of the serial dependence would be smaller or even absent. This effect was present for arousal but not for valence. The serial dependence for arousal was significantly lower in Experiment 2 (3.0) compared to Experiment 1 (4.8),${\mathit{t}}_{96}=2.09$,$\mathit{p}=0.039$; however, the serial dependence for valence was significantly higher in Experiment 2 (6.4) compared to Experiment 1 (2.7),${\mathit{t}}_{96}=3.55$,$\mathit{p}\mathrm{<}0.001$.

4. General Discussion

In the present study, we investigated whether serial dependencies for emotional stimuli transfer from one modality to another. In Experiment 1, we reported a positive serial dependence for valence and arousal regardless of the stimulus modality on two consecutive trials. In other words, the rating for both characteristics (valence and arousal) on a given trial was higher when the rating on the previous trial was high than when the rating on the previous trial was low. In Experiment 2, for the valence rating, we replicated these findings, even though participants did not rate the emotional stimuli on the previous passive trial, indicating that an explicit emotional judgement is not a prerequisite for observing a positive serial dependence. However, with respect to the arousal rating, we observed a positive serial dependence when the stimulus on the previous passive trial was visual, but not significantly so when the previous stimulus was auditory.

The positive serial dependence can be explained well by a lingering (or averaging) effect in (parts of) the emotional network: an emotional stimulus results in a residual activation when the stimulus disappears, so that the subsequent emotional response becomes a mixture of this residual response and the emotional activation caused by the new stimulus, explaining why the rating for a subsequent stimulus is higher when the residual activation (caused by the previous trial) is high than when it is low (i.e., a positive serial dependence). The amygdala is the first brain structure where such a lingering effect could take place as this subcortical brain area processes emotions and typically receives information from all sensory modalities (LeDoux, 2007; McDonald, 1998). Furthermore, it is known that this subcortical brain region responds automatically to emotional stimuli (Arnellet al., 2007; Bannermanet al., 2012; Dianoet al., 2017; Mostet al., 2005; Tamietto & De Gelder, 2010), which can explain why we observed a similar positive serial dependence for emotional stimuli, even though no explicit emotional judgment was required on the previous passive trial (Experiment 2) and hence no higher-order structures needed to be involved to do the task. This indicates that residual activity in the amygdala alone is enough to cause the serial dependence effect or that the full emotional network is activated even if the task is only to passively observe the stimulus. A question that remains is why we did not observe a significant positive serial dependence for the arousal rating when the previous emotional stimulus was passive and auditory.

An explanation for the absence of a serial dependence with respect to the arousal rating after an auditory emotional stimulus may be the nature of the stimuli that were used. First, it is important to note that the participants rated the auditory and visual stimuli differently both in terms of arousal and valence, making it tricky to compare the different conditions. What might be even more important than the overall ratings is that the spread of the ratings differed substantially between the auditory and the visual stimuli. That is, the ratings were rather similar for the auditory stimuli, and more diverse for the visual stimuli (compare the left panels with the right panels in Fig. 5). As a result, the emotional response on the previous trial and thus the residual activation on the current trial (causing the positive serial dependence) varied to a lesser extent after an auditory stimulus than after a visual stimulus, and failed to reach significance in Experiment 2 where the amount of data in statistical analysis was reduced by half compared to Experiment 1. A recent study by Van der Burget al. (2019) corroborates the notion that the spread plays a significant role in the magnitude for serial dependencies. Van der Burg and colleagues observed a positive serial dependence when participants judged the attractiveness of faces, and importantly, the magnitude of this effect scaled linearly with the attractiveness rating on the preceding trial, suggesting that more extreme ratings also cause stronger serial dependencies. In line with this, a study investigating odour perception reported that odours of more extreme (either positive or negative) valence evoke significantly higher amygdala activation than neutral ones (Winstonet al., 2005). With regard to the present study, we therefore propose that the reduced serial dependence (and absence of an effect) after an auditory stimulus is most likely due to the small spread of the ratings, and is presumably unrelated to the sensory modality on the previous trial.

Numerous studies have shown that the task from trial to trial is an important factor for observing a serial dependence (Van der Burget al., 2019). For instance, Van der Burg and colleagues reported a positive serial dependence for facial attractiveness when participants rated a face on each trial, but not when participants judged the gender of a face on the previous trial. In contrast, the present study yielded a positive serial dependence after an active trial, but also after a passive trial. The presence of a serial dependence after a passive trial suggests that (parts of) the emotional network were activated in a rather automatic, bottom-up fashion, in a similar way for visual and auditory stimuli, and regardless of the task from trial to trial (see also Fornaciai & Park, 2018; Van der Burget al., 2013b, 2018). What is striking though, is that the overall positive serial dependence for valence was significantly larger in Experiment 2 than in Experiment 1. This difference across experiments is contrary to what one might expect as the affective appraisal task in Experiment 1 required the activation of the full emotional network while this was not a prerequisite for the passive observing task in Experiment 2. One might expect residual activity to be similar or smaller (but not larger) for a task that requires only partial activation of the emotional network. A feasible explanation for this discrepancy is that the ‘overall’ serial dependence in Experiment 1 actually reflects two serial dependencies: one positive serial dependence for processing emotions and a second one reflecting a negative serial dependence for the motor response (Leeet al., 2020; Zhang & Alais, 2020), whereas the ‘overall’ serial dependence in Experiment 2 only reflects the positive serial dependence for emotions. As a result, the overall serial dependence (which may reflect the sum of two processes, see e.g. Alaiset al., 2017) becomes larger when the participants do not respond to the previous trial (Experiment 2) than when participants respond to the previous trial (Experiment 1), as in the latter condition a negative serial dependence is added to the positive serial dependence for emotions. The presence of two serial dependencies is consistent with the idea that serial dependence operates at multiple processing stages (Kiyonagaet al., 2017), like perception (Cicchiniet al., 2017; Fischer & Whitney, 2014), memory (Makovski & Jiang, 2008) and decision (Fritscheet al., 2017). Although this explanation is possible, one must keep in mind that there were a couple of differences between both experiments. Although we used the same participants and the same stimulus sets, the actual stimuli used for the active trials in Experiment 2 were not identical to those used in Experiment 1. Furthermore, the interstimulus interval between two successive trials was not the same, since it depended on the reaction time in Experiment 1, while the EmojiGrid was replaced by a 1-s blank interval in Experiment 2. We do not believe that these factors play a crucial role in our findings, but it is important to keep these differences in mind.

The present study is not the first study reporting intertrial effects between different sensory modalities. Indeed, the modality switch effect (MSE; Spenceet al., 2001) refers to the increase in reaction time that occurs when switching from one sensory modality to another unpredicted sensory modality. Thus, in the case of the MSE, there is a cost associated with switching from one modality to another, whereas in the present study there is a carryover effect. We believe that these effects reflect different processes, as indicated by a number of differences in characteristics of the effect and conditions under which they occur. First, switch costs typically occur when participants suddenly switch from a predicted to an unpredicted modality, whereas in the present study, the stimulus modality was always unpredictable (as both the auditory and the visual stimuli had an equal probability to occur). Second, our intertrial effect is not defined by the stimulus modality but by the emotional context of the stimulus, regardless of the stimulus modality on the current and previous trial. Taken together, we believe that the present finding of serial dependency of emotion refers to another effect than the MSE.

Emotions have a huge impact on our daily behaviour. For instance, the emotional state has an impact on what we consume (see Watson & Spence, 2007, for a review), on how we make decisions (Vohset al., 2007), and how an athlete performs during a match (Hanin, 2000). The present findings have important implications, since we show that a particular emotion is in fact not only triggered by an event (or stimulus) at a given moment in time, but reflects multiple emotions over time. Indeed, in a recent study, we found evidence that the emotion on a given trial depends on the rating up to three trials back when participants rated food images using the same EmojiGrid as in the present study (Van der Burget al., submitted). Such a lingering effect was also observed when participants rated the attractiveness of faces (Van der Burget al., 2019) and when participants made a temporal judgement on every trial (Van der Burg & Goodbourn, 2015; Van der Burget al., 2013b). Moreover, we have shown for the first time that emotions, even though they are not related, carry over from one event to another, and that this occurs regardless of the sensory modality in a rather automatic fashion. This knowledge can be used in many settings. For instance, one can increase the emotional experience in a restaurant by playing some pleasant classical music as soon as costumers enter the restaurant. However, what is pleasant is largely subjective, so one must be sure that the emotion triggered is indeed a positive experience for everyone. An intriguing question for the future is to determine how the delay between two successive events relates to the positive serial dependence.