In the digital age, many crimes occur within or around a digital device. These crimes are witnessed by the digital device. Sometimes incidents are caught on camera, but many times they are only decipherable by studying the artefacts left behind. In the digital space, various artefacts have been utilized as forensic evidence. These include computer system data such as login credentials, network traffic data and software usage metadata [1].

Meanwhile, human-computer interaction is fast moving from being an interaction between two disconnected entities into a more merged and symbiotic computational connection. More smart devices are being created every day to couple tightly with the human body. While this approach may increase convenience, the impact of adverse digital incidents is also likely to increase. This calls for improved methodologies geared towards safer digital spaces.

Broadly, there are two aspects in which safer digital spaces could be created: one would involve methods of real-time monitoring and adaptive response to threats, and the other is availability of digital metadata with evidential value. Such metadata can assist in solving open questions in forensic investigations to support justice and law enforcement functions.

There are many forms of forensic methodology, each with its own foci and applicability spectrum but none has proved perfect for exclusive use [2]. Hence, digital investigations benefit more with every additional form of acceptable evidence that is available.

In the past psychophysiological data was not typically available outside of clinical or specialized settings. Perhaps this explains why this form of data has yet to be widely explored as a form of digital evidence. Today however, many end user tools are capable of collecting and storing such data within their regular context of usage. Examples include fitness monitors, smart eyeglasses, smart clothing and even gaming headsets among numerous other smart body items. Even smartphones and other non-wearable devices are now able to collect human-generated artifacts such as motion, pressure and eye gaze data.

In light of that change, it is no longer far-fetched to study applications of human psychophysiological signals as a potential source of markers of digital incidents. Finding and harnessing such markers would yield an additional form of digital evidence of incidents, as well as a potential feedback channel revealing threats in the digital environment and triggering appropriate responses in a timely manner. Such threats could include cyberbullying and insider attacks.

There has been some work aimed at securing digital spaces. As an example, Mondal and Bours [3] define a continuous identification model based on hand swiping movements to continually verify that the authorized user is the one using the touch screen of a mobile device. They envision adding a forensic component if the model is used within a closed system that allows it to attempt to identify any intruder.

From a forensic perspective, an example usage of psychophysiological data involved the application eye gaze tracking to determine if witnesses recognized evidence that was in front of them [11].

Despite the potential to be a rich source of evidence data in digital investigations, psychophysiological signals have remained largely unexplored in this regard.

The human is a key component of a computing network with high emotional responsiveness to the environment. This responsiveness manifests as psychophysiological changes occurring even when no physical emotional reaction is observable. Further, psychophysiological signals are hard to manipulate and so they are likely to provide a true reflection of digital incidents. Therefore, analyzing recorded signals for structural changes, could reveal information regarding the state of other components in the computing network during the same time period. Identifying and harnessing this information resource can yield evidence towards digital investigations, as well as enable innovations that support a safer and more secure computing environment.

The goal of this research work is to investigate how various psychophysiological signals are affected by response-evoking properties of stimuli—such as novelty, salience, aversiveness, and the element of surprise. This involves investigating the timing of onset of interaction with a digital event, duration of interaction and timing of the end of such interaction.

Recording psychophysiological data allows for circumventing conscious decision-making through probing involuntary feedback channels [12]. Psychophysiological change is a manifestation of emotional reaction to events [4]. There are various psychophysiological feedback channels of emotional response. Examples include electrodermal activity - measuring skin conductance responses, electromyographic activity - facial muscle movements, electrocardiogram activity - heart pumping activity and electroencephalography - brain electrical activity.

3.1Emotion in HCI

Emotion is a consequence of human appraisal of a situation and is a reflection of the resulting affect [4]. There can be various response eliciting properties in a stimuli. These include novelty, significance, salience, surprise, intensity, arousal [60].

Emotion-based studies have been done under various analysis approaches. One major dichotomy is between theDiscrete vs Dimensional approaches. Discrete emotion [5] theories analyze emotion through its manifestation in physical expressions and functions drawn from a discrete and limited set e.g. interpretation of facial or hand expressions by function. Dimensional theories [6] on the other hand do not evaluate emotions as discrete and limited but rather as a large range of emotional states within a defined dimensional space (e.g. a one dimensional model defining motivation towards action such as approach-avoidance model).

For this study, we apply the dimensional evaluation model and are concerned with intensity of responses triggered by those emotions rather than categorization of the specific type of arousal or valence. This would be ideal for analyzing signal when the source person is not available to be observed or questioned—as would be common in an investigation context.

In turn, there are various dimensional approaches in studies involving psychophysiological response. These include models such as Valence-Arousal [6], Approach-Avoidance [7], and Threat-Challenge [8]. For this work, we apply the Valence-Arousal model. In theValence - Arousal model, emotion is considered in terms of its location within the valence-arousal dimensions quadrant i.e. high arousal positive, high arousal negative, low arousal positive or low arousal negative e.g. anger may be evaluated as highly negative and highly aroused. Valence-Arousal can be calibrated using pictures from the International Affective Picture System (IAPS) which are standardized and pre-rated for valence-arousal [9]. These dimensions have been found to show considerable consistency across cultures [10].

3.2Psychophysiological Feedback Channels: Basis of Markers of Response to Stimuli

Electromyography in Corrugator Supercilii

Overview of Mechanism

Electromyography (EMG) measures activity of muscles. Muscles are the bodily tissue that generates and transmits force [13]. The electromyogram (EMG) is an electrical signal generated following muscle contraction. This signal reflects the electrical and not the mechanical, events of the contraction [14]. Cardiac and skeletal muscle groups are striated i.e. they are composed of bundles of thin fibers known as fibrils [15]. Each striated muscle is innervated by a motor nerve through which neural signals are delivered. There neural signals—muscle action potentials (MAPS)—are responsible for all actions of striated muscles [16].

EMG voltage changes as a result of multiple muscle action potentials (MAPs) across many muscle fibres within several motor units rather than a direct measure of muscle tension, contraction or movement. Hence, the signal measured using surface electrodes is attributed to muscle activity in a given muscle region or site rather contraction of a specific muscle [13]. Specific location of muscle contraction is difficult to determine due to the close proximity in the arrangements of striated muscles and the non-specificity of surface electrodes [13,17]. EMG is measured using surface electrodes due to their non-invasive nature and because psychophysiological research questions are concerned with muscle sets rather than motor units within muscles. Surface EMG measurement detects ongoing muscular contraction in situations where simple observation by eye is too imprecise [13].

EMG Markers of Response to Stimuli

Subtle psychological processes often cause EMG activation without any accompanying visually perceptible actions or visceral changes [18,19]. For example, while muscle activation will accompany facial expressions, muscle activation can also occur without the occurrence of any overt facial distortions, such as when activation is weak or fleeting or suppressed [20].

Negative sensory stimuli and mild negative imagery cause increased activation over the brow region also known as corrugator supercilii. This upper face site is the muscle region that draws brows inward and downward, sometimes forming vertical wrinkles [13]. Activity over the brow region (corrugator supercilii) varies inversely as a function of affective valence of stimulus i.e. negative affect such as disgust leads to increased activation over the brow region [13]. Several studies have shown increased EMG activity in the corrugator supercilii region when negative imagery or sensory stimuli is experienced [21]. As an example, when participants with depression were asked to imagine unpleasant experiences they displayed increased EMG activity in the corrugator supercilii region [24]. Non-depressed participants showed similar patterns although at a lower scale. Other negative emotions that have been found to increase EMG activity at the brow region include anger, fear, sadness, surprise [22], and disgust [23]. These studies suggest that the corrugator supercilii may be a suitable EMG feedback site when testing for high valence and negative affect.

Decrease in activity of the corrugator stimuli has been observed following positive stimuli such as presenting participants with pictures of smiling faces [26]. Some studies indicate that corrugator EMG may even suffer from interference when some non-negative emotion is faked.

In a deception study—using EMG and facial action coding—of individuals who pleaded for return of their missing relatives in televised recordings, deceptive participants showed reduced contraction of the corrugator supercilii compared to honest participants [25]. Faces of these individuals also showed masked smiles while they tried to put on sad looks. Half the individuals in the study had been eventually convicted of murdering the persons prior to pleading with them to return home. This may be an indication that general EMG activity can be interfered with by feigning an emotion. or simply that EMG is not a good indicator for deception. However, there is not much literature studying deception with EMG.

The baseline is used to determine the onset of stimuli-induced affect in EMG measurements. The true physiological baseline of EMG is zero [13]. In this ideal case, the lowest empirical baseline would be the level of noise in the recording equipment. However, muscles are unlikely to be completely at rest, especially in a laboratory setting, as the participant is never completely relaxed. Therefore, the baseline is considered to be the EMG activity that exists in the absence of experimental stimuli [13]. Once the baseline has been determined, changes in signal frequency can be interpreted to be signalling the ongoing response to stimuli.

Signal amplitude is commonly used as the dependent variable in psychophysiological experimentation [13]. Counting or averaging the EMG peaks in amplitude or tallying directional changes or signal crossings can be used to gauge EMG activity provided a high sampling rate is used [27]. However, some researchers consider Integrated EMG signal—the total energy of an EMG at a given time—to be a more meaningful way of measuring of overall muscle contraction than by counting or averaging amplitude peaks [28,29,30]. There are various techniques used in deriving the integrated EMG e.g. computing the arithmetic mean of a rectified and smoothed EMG [13].

Electrocardiography (ECG)

Overview of Mechanism

ECG is a measure of heart rate variability assessed using various metrics in the time or frequency domains. The time and frequency domain methods are complementary ways of characterizing the same sets of variances [31]. Time domain methods include measures of variance of heart periods and of their distributions as well as geometric methods based on heart period distributions [32]. Measures include standard deviation of the normal beat-to-beat intervals. Frequency domain methods decompose the heart period variance into frequency bands [32,33]. One example method that can be used to approximate any periodic time-varying waveform is the fast fourier transform, by which the summation of a finite set of pure sinusoids of differing amplitudes can be computed [31,32,33].

High frequency heart rate variability is associated with variations in parasympathetic control in respiration [34,35]. This variability is widely used as an index of vagal heart control [34,35,36,37,38]. The basal heart rate variation (Respiratory Sinus Arrhythmia - RSA) can be influenced by factors such as age, activity and posture. Hence RSA variability across subjects may not offer valid comparison in vagal control without controlling for those factors. The within-subject variability may be more valid as a vagal control measure provided properties such as posture and activity are taken into account [31]. Low frequency variability is associated with both autonomic branches and hence not regarded as a pure index of either [31], even though low frequency variability can be useful in measuring baroreflex action and cognitive workload [39,40]. This low frequency bands have also been applied in indexing of autonomic balance which is evaluated along the sympathetic-parasympathetic spectrum [31,41].

Cardiovascular measures have been used in study of arousal, stress, emotion and cognitive processes [31]. For example, high frequency heart rate variability has been found to indicate attentional capacity and performance [45]. Heart rate variability has also been found to decrease with increased workload [31,40,42]. Heart rate changes have also been found to distinguish between tasks involving external stimuli such as listening to noise, and internal stimuli such as attention to information processing [46]. RSA/RSA reactivity can account for a third of the variability in a between-subjects psychomotor vigilance task [31,50]. Pre-task RSA has been found to predict performance in cognitive tasks requiring short-term memory [31,51]. Baroreflex measures—reflecting the blood pressure control activity of baroreceptors arising in the arteries—have also been found to be highly responsive to psychological events such as mental effort and stress [42,43,44]. Stress has been found to reduce baroreflex gain.

ECG Markers of Response to Stimuli

Heart rate variability is highly responsive to increased arousal, workload and mental effort. Reduction in HRV is associated with increased arousal [31], increased workload [40,42], and increased mental effort [45]. High frequency heart rate variability can predict the level of attentional capacity and performance in a person [45].

The “heart rate” refers to the number of heart beats per minute (bpm). Activity of the heart occurs in cardiac cycles with each consisting of the events between one heartbeat and another. In each cardiac cycle, there is a period when the heart does not pump blood, and one when the heart pumps; These are referred to as the diastole and systole respectively. The diastole and systole represent the blood pumping activity of the heart. This pumping action helps maintains the flow of oxygenated blood into the lungs and the rest of the body [31].

Blood flow is regulated by intrinsic mechanisms arising locally within cardiac tissue as well as extrinsic ones arising from hormonal or autonomic effects. blood flow can be altered by local mechanisms which adjust tissue structure to meet the need (e.g. when a cancerous tumor occurs, blood vessels will increase to meet the increasing need for additional blood flow). Interactions with extrinsic mechanisms such as the actions of autonomic neurons are also able to cause activations that affect aspects of cardiac function [52,53,54,55,56]. These activations can alter the interval between one heartbeat and the next and hence change the heart rate. The parasympathetic stimulation is more predominant than sympathetic stimulation in control of heart rate [34]. Heart rate has a generally linear relationship with parasympathetic activity while it has some non-linear relationship with sympathetic activity [31].

The QT interval of the ECG represents the time from ventricular excitation until the return to resting state ranging between 200–500 ms. The intervals are shorter with higher heart rates. The QRS peak which lies between the Q and T points corresponds in particular to the timing of the invasion of the myocardium which is the peak response to the electrical activation. This segment lasts about 100 ms unless there is a block in the branches within the conduction system resulting in a prolonged interval [31]. Hence a change in measured ECG signal, would be observed within 500 ms from the onset of the stimulus that caused it.

Heart period is the time in msec between adjacent heart beats, measured between successive R spikes in the ECG and it is a value reciprocal to heart rate. The two values can therefore be converted into each other and neither is dominant as the primary metric for cardiac measurement. In spite of that, heart period has been recommended for circumstances where a strictly linear relationship with autonomic inputs is desired [31]. A change in activation in either autonomic branch will lead to about the same change in heart period regardless of the baseline [57,58]. Due to this mostly linear relationship, the use of heart period has been recommended as opposed to heart rate as a metric when changes in heart period are anticipated from autonomic responses as in psychophysiology experimentation [31,57].

Heart rate has a non-linear relationship with autonomic events and its measurement may often suffer the effects of accentuated antagonism between the two autonomics branches [31,59]. Heart period appears to be less disturbed by these interactions and hence is also recommended for studies where cardiac function may vary widely such as under varied experimental manipulations [31,57].

Heart period is often converted to and represented as heart rate, although it is sometimes used on its own [31]. Change in heart period reflects the autonomic effect of response to a stimulus. This indicates that the disturbance in heart period would continue to persist while the stimulus continues to elicit responses that may correspond to the duration of the stimulus.

Electrodermal Activity (EDA)

Overview of Mechanism

EDA measurement reflects the level of electrical resistance generated by the skin. Resistance has been long known to decrease in response to sensory stimuli [61]. This means that the skin becomes a better electrical conductor in the moment after receiving stimuli. EDA can be measured endosomatically or exosomatically—internal or external measurement respectively [60]. In this section we reference exosomatic EDA measurement which is more suitable for this work. Using this method, the change in resistance is observed by passing a small current onto surface electrodes on the skin during the presentation of the stimuli [60].

The mechanism underlying the changes in skin surface resistance is the eccrine sweat glands whose primary function is thermoregulation of the body by evaporative cooling [60]. However, eccrine glands have been found to be responsive to physiologically significant stimuli. The eccrine glands on plantar and palmar surfaces in particular, have been thought to be more responsive to physiological stimuli that to heat, possibly due to the high gland density in those regions [60,62,63].

Innervation in the sweat glands arises predominantly from fibers in the sympathetic chain [63]. As such, specific conductance response (SCR) has been found to correlate highly with activation in the sympathetic nervous system [64]. The amygdala in particular has been found to show high levels of activation when stimuli elicited SCR [65,66].

As sweat level increases in a given sweat duct—columnar components of the gland that open onto the surface of the skin and act as variable resistors—resistance in the sweat duct decreases. Hence, the change in sweat levels and the resulting changes in resistance lead to the changes in the measured EDA. The measure commonly used for controlled experiments is the amplitude of skin conductance response (SCR) which is the amount of increase in conductance from the onset of the response to the peak [60]. This measure is a part of the skin conductance level (SCL) which is the overall tonic level of conductivity of the skin. In some cases, SCL is a more suitable measure than SCR. An example is SCL usage in studies of continuous situations without specific stimuli for which a SCR can be measured [60].

When experimental stimuli is repeated, an average SCR size can be computed—either magnitude or amplitude—to represent the average response value across trials. A magnitude average includes trials that returned no responses, while an amplitude average includes only non-zero trials. The former measure is commonly used but both have applications where they are suitable [60,67].

EDA has been found to respond to many types of tasks and it is argued that SCRs on their own are hard to link to a specific psychological response to stimuli such as anxiety or anger [68]. However, knowledge of the stimulus conditions coupled with carefully controlled experimental paradigms enables such inferences to be done. Another way that SCR to stimuli property relationships have emerged is through consistencies occurring between concurrently observed brain and skin activations [60].

There are a number of other disadvantages with EDA. First, EDA measures often suffer interference in the way of superimposed responses [67,69] (i.e. the size of a response is a function of time since the previous response and size of that response if superimposed). Hence, EDA studies typically require long interstimulus intervals of at least 20–60 s. Another problem is inter-individual variance due to extraneous differences between individuals [60]. Range correction was initially proposed as a solution, where each individuals range is computed separately and their response quantified within that range [70]. This method is however unsuitable for some situations e.g. comparing individuals in different ranges [71]. A different solution to this problem is the use of within-subject standardized scores to adjust for individual differences.

Another problem with SCR-based studies are that SCRs can be impacted by habituation as the stimulus becomes more familiar. This causes decline and gradual disappearance of the SCRs. Various measures of habituation can be computed including trials-to-habituation count, decline of SCR across trials as assessed in interaction effect in analysis of variance, or regression of SCR magnitude on the log of trial number [60,72,73].

Due to the limitations of EDA, it is recommended to use it as one of multiple measures in experiments. That approach enables the researcher to tap into EDA’s general utility as an indicator of arousal and attention, while also being able to get specific about psychophysiological state via a better differentiator, such as heart rate [60].

EDA Markers of Response to Stimuli

Increased SCRs have been observed when parts of the brain are involved in effortful activity [60]. As an example, SCR has been found to increase at the decision-making stage prior to risky or bad decisions in gambling tasks [74]. Thermal pain stimuli also showed increase in SCR [75] matching with increased responses in the neural regions that respond to pain including the thalamus and ACC. During rest, SCRs increased in line with brain activation in the ACC—anterior cingulate cortex which is associated with consciousness [76].

Other examples of SCR eliciting properties of stimuli include: novelty, unexpectedness, aversiveness, salience and emotional significance [60]. For instance, a discrete stimuli that elicits a response to the significance (salience) property in stimuli is the guilty knowledge test also referred to as Concealed information Test [77].

SCRs that are elicited by any non-aversive stimuli are initially considered to be an orienting response (OR) [72]. A minimum amplitude threshold is used to determine when SCRs can be linked to the properties of the specific stimuli. This minimum is commonly set between 0.01 and 0.05 μS [60]. Further, a minimum latency window is also set to prevent counting of spontaneous responses—non-specific skin conductance responses (NS-SCRs) e.g. as might result from bodily movements—as responses to experimental stimuli. Typically, SCRs beginning inside 1–3 or 1–4 s windows from the stimuli onset are considered as elicited by that stimuli [60].

Elicited response is determined against the baseline—set between 0.01 and 0.05. The size of elicited SCR often ranges between 0.1 and 1.0 microvolts with variations arising from environment [60], methodology and individual differences [78].

Where stimuli presentation is not discrete such as video games or in situations continuing over long periods, SCL and frequency of NS-SCR measures are preferred over SCR because the tonically varying levels of arousal. Both measures will show change between resting level to anticipation level and then to action level for almost any task [60].

In continuous tasks, SCL has been found to increase in response to both external stimuli such as loud sound, and internal stimuli such as information processing tasks [46]. SCL and frequency of NS-SCR have been found to be responsive to various continuous psychological stimuli situations including: anticipation and performance of any task [46], task switching and video gaming [60]. This observation indicates that tonic EDA is applicable for indexing processes related to energy regulation and it has been interpreted that the EDA responses are caused by effort of allocating information resources [60] which increases autonomic activation.

These measures also show an increase during non-task related continuous stimuli such as fear, anger and suppression of facial emotion display during viewing of a movie. Social stimulation through emotions such as stress and anxiety also caused an increased SCL and frequency of NS-SCR [60].

Electroencephalography (EEG)

Overview of Mechanism

Nunez and Katznelson [89] discuss that EEG signals occur and are recordable without the need for any deliberate stimuli application. However, changes in EEG signal, occurring due to response to a specific stimuli are referred to as evoked potentials.

Brain responses to stimuli are quantified by measurement of EEG amplitude or energy changes [90]. EEG signals can respond to various types of stimuli including visual stimuli, somatosensory stimuli and motor imagery [91]. Examples of media used to present visual stimulation include the Snodgrass & Vanderwart picture set [90] which has been found to induce highly synchronized neural activity in the gamma band [92]. Lists of acronyms have also been used as stimuli for Visual Evoked Potentials [91].

EEG has been utilized for brain-computer interfaces (BCI) in medical and non-medical settings. Several applications have been derived including: automated diagnosis of epileptic EEG using entropies [93]; automated drowsiness detection using wavelet packet analysis [94]; EEG-based mild depression detection using feature selection methods and classifiers [95]; neuro-signal based lie detection [96]; authentication [90] and continuous authentication.

EEG/ERP Markers of Response to Stimuli

ERPs represent EEG signals resulting from exposure and response to a particular stimuli. They run from 0 to several hundred milliseconds [79]. These waveforms are characterized by sets of positive and negative peaks labelled P and N respectively. These labels indicate the direction of polarity following a stimuli exposure and usually have a number specifying the ordinal position or the latency of the peak [79].

ERP Peaks are different from ERP components which are the changes that reflects a specific neural or psychological process [80]. ERP components arise from electrical activations in the brain and are then conducted through the skull and onto the scalp where they are measured using scalp electrodes [81]. The peaks in an ERP waveform do not directly translate to an underlying ERP component, although early peaks reflect sensory responses while later peaks represent motor and cognitive responses. To obtain ERP components, various statistical procedures can be used. Two major methods used to isolate ERP components are principal component analysis (PCA) and independent component analysis (ICA). PCA finds the components that individually account for the largest variation in activation while ICA finds those components that are maximally independent [79].

Various ERP components can be obtained from EEG depending on the nature of stimuli. These include the P3 component, Mismatch Negativity (MMN), N2 posterior-contralateral (N2pc) component and several others. P3 amplitude depends on the probability of occurrence of the target stimulus—as defined by the task—amongst the more frequently occurring non-target stimuli [82]. The P3 component can be found by subtracting the amplitude of the rare stimuli from the frequent one. The onset time of the resulting difference wave corresponds to the reaction time needed to perceive and categorize a stimulus [80].

When participants ignore the stimuli, no P3 wave is elicited by the unexpected [80]. However, surprising sounds typically elicit a negative potential, from 150–250 ms, even while the stimuli is ignored. This negative potential trend is known as mismatch negativity (MMN) [79,83]. MMN is known as pre attentive or automatic potential due its arising in task-unrelated stimuli [79]. This potential can be calculated as a difference wave in a similar manner as the P3 component. This potential has been applied for assessing processing in locked-in individuals, such as preverbal infants [84].

N2 posterior-contralateral (N2pc) component occurs during a participant’s focused attention to a target while multiple stimuli are presented laterally on each trial. It appears at the posterior electrode 200–300 ms after onset of each stimuli array [85]. Deflection at the left hemisphere electrode reflects when the target is located in the right hemisphere and vice versa. [79]. N2pc can be used to infer that an object has elicited a shift in attention [86]. The timing of N2pc can help to compare how fast attention shifts for different types of targets [87] and among different participants or groups [88].

4.1Task Description

Participants will play an on-screen game during a session in which their psychophysiological activity will be recorded. Various sensors will used to measure signals of this activity. The specific measurements will include electrodermal activity – to record the skin conductance responses, electromyographic activity to record the facial muscle movements, electrocardiogram activity to record the heart pumping activity and electroencephalography to record the brain electrical activity.

The stimuli of interest will consist of unexpected interruptions that we introduce into the participant’s session. These interruptions are selected for their computer security implications—participants will be debriefed with full information after the task. We will be using plain non-security events as a control.

The purpose of this study is to assess if the signals collected by the sensors during the session contain any information that can be useful as markers of the events that occured. If these events repeatedly create significant structural changes in physiological signal, then such signal could be examined for specifics such as onset, duration and cessation of the event.

This would then open the way to studying how best to harness such markers towards various applications e.g. as a viable source of digital forensic evidence.

4.2Selection of Psychophysiological Measures

In selecting physiological measures to base the study on, considerations included ease of obtaining a signal reading, non-intrusiveness, cost, and the reliability of signal to reflect the changes in psychophysiological state. In addition, the significance of psychophysiological states have been found to draw from activity across different feedback channels rather than to reflect a discrete response domain [31,47,48,49].

For this work, ECG, EDA, EMG, EEG were selected. All can be measured with relative ease and measurements are digitized automatically. A combination rather than a single one of these methods is often better at closely tracking both sympathetic nervous system (SNS) and parasympathetic nervous system (PNS) responses.

4.3Selection of Stimuli

In order to correlate physiological measures to specific emotional states, it is necessary to select stimuli that create conditions for the participant to be able to reach the necessary emotional state during the task e.g. to unambiguously elicit disgust or amusement or anger. At the same time, the stimuli should not cause or allow for conditions that may hinder response e.g. by eliciting shame when the study is not concerned with shame or by allowing unusual levels of boredom.

For this study, participant sessions will be interrupted by various unwanted events, which will constitute the stimuli of interest. It is intended that response to negative interruptions will mimic the affect caused by events typical as part of cyber incidents.

Between the interruptions, participants will play an on-screen game. Therefore they will perform a cognitive task as a distractor and in between stimuli. The purpose of the cognitive task is to ensure that the participant is mentally engaged with the digital device environment. They study task will require them to interact with the event, and that will enable us to link physiological responses occurring at the same time to these events.

However, we are also interested in recording the timing of their response in order to determine where to find the corresponding markers if any within the recorded signal. Timing will also help us infer when and how psychophysiological signal reflects response to the stimuli by comparing the onset of the physiological response markers, mouse or keyboard response markers and onset of exposure to stimuli.

Therefore, it is important to reduce the amount of extraneous delay between the onset of the participant’s mouse or keyboard response and the other two pieces of onset timing information. Engaging the participant in a moderate cognitive task within the stimuli environment ensures that they are alert and ready to begin interacting with the stimuli as soon as is natural for them.

The gaming task in particular is an ideal cognitive task because no memorization is needed. Each gaming component is resolved while it is visible, and nothing is lost during the interruptions. Hence as soon as the stimuli exposure begins, the participant’s cognitive resources are released to the stimuli.

4.4Validating Responses

Even when the stimuli are valid for the task, the responses and their intensities will differ by participant. In addition, different responses may be combined. Verification and calibration methods can include coding of facial behavior by automated systems or cultural informers as well as self-reporting by the participants.

In this study, the task requires a mouse or keyboard response to stimuli, allowing us to estimate the timing when responses began. This is coupled with a repeated measure strategy to allow for comparison of signal patterns. Each stimuli will be presented multiple times during the course of the study. Further, we do not classify the responses in discrete emotion terms—e.g. anger, frustration, surprise—but rather as a measure of intensity of valence or arousal affect.

For comparison, there will be stimuli that will tend towards neutral in nature e.g. a call to rate the game or sign up to a newsletter about the game and there will be interruptions of a security nature with implications unknown to the participant during the task—they will be debriefed with full information at the end of the session. The study will assess the signals collected at the sensors during the session, for any structural information corresponding to the interrupting events that occurred during the session. If there is such information, and responses to neutral events present structural information that differs from that of responses to non-neutral events, then the latter can be regarded as psychophysiological markers of those types of events.

These markers can then be utilized to examine for specifics of when and how long such events occured.

The human factor is a key component of the computing network, and human emotion is highly responsive to its environment. Hence, psychophysiological signal activity can hold a lot of information about an individual’s experiences while using a computing device. This form of metadata while largely unexplored for this purpose, is particularly promising as it is preconsciously controlled. This makes it less susceptible to human decision-making errors and deliberate tampering than other forms of computing metadata.

With cybersecurity becoming a concern in many contexts, psychophysiological markers of digital incidents can be useful as forensic evidence. Such markers could also be used in designing of tools that help create safe digital spaces, by triggering appropriate controls to protect individuals. Further these markers could be used to manage insider threats also by triggering appropriate controls to secure digital resources that are in use by a potentially rogue insider.

This work describes the theoretical background and the study aimed to examine the relationship between digital events with high valence or affect properties, and the structural properties of recorded psychophysiological signals occurroccurringing simultaneously.

We seek to determine if the signals recorded during these events contain any information that can be useful as markers of the events that occured. If these events repeatedly create significant structural changes in psychophysiological signal, then such signal could be further examined for specifics such as onset, duration and cessation of the event. This would in turn lead to studying how best to harness such markers towards various applications (e.g. as a viable source of digital forensic evidence).