Repository files navigation

Emotional Dialogue Act data contains dialogue act labelsfor existing emotion multi-modal conversational datasets.

Dialogue act provides an intention and performative function in an utterance of the dialogue.For example, it can infer a user's intention by distinguishingQuestion,Answer,Request,Agree/Reject, etc.and performative functions such asAcknowledgement,Conversational-opening or -closing,Thanking, etc.

The aim is to enrich the existing multimodal conversationalemotion dataset that would help advancing conversational analysisand help to build natural dialogue systems.

We chose two popular multimodal emotion datasets:Multimodal EmotionLines Dataset (MELD) andInteractive Emotional dyadic MOtion CAPture database (IEMOCAP).

MELD contains two labels for each utterance in a dialogue:Emotions and Sentiments.

• Emotions -- Anger, Disgust, Sadness, Joy, Neutral, Surprise and Fear.

• Sentiments -- positive, negative and neutral.

IEMOCAP contains only emotion but at two levels:Discrete and Fine-grained Dimensional Emotions

• Discrete Emotions: Anger, Frustration, Sadness, Joy, Excited,Neutral, Surprise and Fear.

• Fine-grained Dimensional Emotions: Valence, Arousal and Dominance.

Our analysis on EDAs reveal association betweendialogue acts and emotional states in natural-conversational language.Visualizing co-occurrence of utterances with emotion statesin the particular dialogue acts, please see the graph below from IEMOCAP EDAs:

See live plots/graphs (thanks to Plot.ly):IEMOCAP Emotion Sankey,MELD Emotion Sankey, andMELD Sentiment Sankey.

or bar diagram (as in the paper):

IE: IEMOCAP, ME: MELD Emotion and MS: MELD Sentiment.

See separated bar graphs in figures:IEMOCAP Emotion Bars,MELD Emotion Bars, andMELD Sentiment Bars.

These graphs can be generated usingread_annotated_..._data.py applications

"Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators"

The article (accepted at LREC 2020) explaining these datasets can be found at -https://www.aclweb.org/anthology/2020.lrec-1.78/

This work has been performed as part of the SECURE EU Project

https://secure-robots.eu/fellows/bothe/EDAs/

• annotated_eda_data/eda_iemocap_no_utts_dataset.csv - contains the EDAs in IEMOCAP data, without utterances.
• annotated_eda_data/eda_meld_emotion_dataset.csv - contains the EDAs in meld data, they are staked (train, dev, test).

In case, if you want to reproduce the python environment used for the entire project,please find theconda_env/environment.yml filewhich can be maneged via conda (anaconda or miniconda).

1. Rundia_act_meld_ensemble.py ordia_act_mocap_ensemble.py for respective data,to generate and calculate the reliability metrics ensemble of the dialogue acts given that all the labels are stored inmodel_output_labels directory.

2. Runread_annotated_mocap_data.py orread_annotated_meld_data.py to read, generate the the graph and and calculate the data statistics

Currently, all the predictions are available in themodel_output_labels directory, hence we skip the following.

3. Rundia_act_meld_annotator.py ordia_act_mocap_annotator.py to annotate the datasets;given that models exists they will be updated soon.

In case, you want to annotate the conversational data,please find thedialogue_act_annotator.py file,which contains a client example for sending the requests to the server APIthat takes care to generate utterance representation and make the predictions.

In case the server is not working, host your own server usingthe repositorybothe/dialogue-act-recognition.

You just need to arrangeutterances into list along with otherlists such asspeaker,utt_id andemotion(it is optional, if you have any other annotation per utterance,or just pass flagis_emotion=False inensemble_eda_annotation() function.).

This API is not fully tested, hence send the requests at your own risk!

However, it is fully automated and hosted viaUni-Hamburg, Informatik web servers.

Try to send, not more than 500 utterances, if it is above that number,server might take long to respond, hence it is recommendedto contact author [bothe] for your query.

Paper explaining the process of dialogue act annotation:

-EDAs:

C. Bothe, C. Weber, S. Magg, S. Wermter.EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators.(LREC 2020)https://secure-robots.eu/fellows/bothe/EDAs/

The original work of datasets:

-IEMOCAP:

Busso, C., Bulut, M., Lee, C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S.IEMOCAP: Interactive emotional dyadic motion capture database.(Language Resources and Evaluation 2008)https://sail.usc.edu/iemocap/

-MELD:

S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, R. Mihalcea.MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation.(ACL 2019)https://affective-meld.github.io/

Chen, S.Y., Hsu, C.C., Kuo, C.C. and Ku, L.W.EmotionLines: An Emotion Corpus of Multi-Party Conversations.(arXiv preprint arXiv:1802.08379 2018)