Repository files navigation

Detection of MBTI-type personality with state-of-art language models

Software version: Ubuntu version 16.04 | Tensorflow-gpu version 1.11 | Keras 1.0.8 | cuda 9.0 | python 3.6.10

Minimum server requirements: 30 Gb CPU, 15 Gb NVIDIA P100 GPU

Step-by-step instructions for setting up the environment:

1. Download libcudnn7-dev_7.4.2.24-1+cuda9.0_amd64.deb, libcudnn7-doc_7.4.2.24-1+cuda9.0_amd64.deb, libcudnn7_7.4.2.24-1+cuda9.0_amd64.deb to your home folder from Nvidia developer website. Please note the versions (the module may not be compatible with any other version)
2. Clone the git repository to your home
3. cd /$repository/setup/
4. Run setup.sh from setup dir with commandsh setup.sh
5. Naviagte back to home dircd ../
6. Runsh requirement.sh
7. Import bash file bysource ~/.bashrc
8. copy mbti9k_comments.csv file (~2GB) to ./data/ (This file can be obtained on request)

Step-by-step instructions for running the codes:

First one needs to setup the config file. Please follow the instructions below file-by-file according to your experiement. You may want to change the config file parameters from script to script to reduce the runtime.

1. cd ./src
2. To quickly check a summary of the data runpython3 data_summary.py. This will output a file in data directory with sentence length for each user.
3. Runpython3 src/data_prep.py to create training samples for both the models.
4. Runpython3 train_bert.py This will fine tune the BERT model for the classification task. It willcreate model.ckpt-NUM files in the model_folder/outputs directory.
5. Runpython3 predict_bert.py. If you set the parameter toFLAG = 'H' then it will prepare a dataset with grouped embeddings for the entire document of each individual user and pass it to the RoBERT model. Otherwise it will run the fine tuned BERT model on a test dataset. Also you need to set theBERT_NUM parameter from theckpt file from step 3. Please refer to the script for other parameters.
6. Runpython3 run_model_roBert.py to run the RoBERT model.

Please note some of the functions in ./utils/ of this repository were taken from Google Research's BERT repository

Results: We achieved 41% accuracy on an 8-class and 43% accuracy on a 4-class classification problem with MBTI type personality detection in our research. We also ran the 4-class classification task with RoBERT for the full document and achieved the best accuracy of 29.8%.

4-Class

8-Class

For 4-class classification the fine-tuning and training BERT-base model took 2 hr 25 mins with 230,428 examples (each of length 150 words). For the same classification task with RoBERT using 5 epochs, batch size of 5 took 2 hr 43 minutes with 7218 samples (whole document). Other computation times for data preparation and other experiments can be found in the log files.

This repository has been developed and maintained by Samrat Halder as a part ofELEN6040: Deep learning Research course offered by Professor Zoran Kostic at Columbia University. For any communication related to this repository please contactsh3970@columbia.edu

This repository has been released under GPL-3.0