Repository files navigation

This repository demonstrates the implementation and deployoment of a facial expression recognition deep learning model using Apache MXNet, based on theFER+ paper by Barsoum et. al..

The repository consists of the following resources:

1. Scripts for data pre-processing as suggested in the paper.
2. Notebook for model building and training, usingApache MXNet.
3. Model deployment for online inference, usingMXNet Model Server

Check out a workingFER+ web application demo, powered by AWS.

# Install MXNetpip install mxnet-mkl # for CPU machinespip install mxnet-cu92 # for GPU machines with CUDA 9.2    # Other Dependenciespip install Pillow # For image processingpip install graphviz # For MXNet network visualizationpip install matplotlib # For plotting training graphs# Install MXNet Model Server and required dependencies for inference model servingpip install mxnet-model-serverpip install scikit-imagepip install opencv-python

Note: please refer toMXNet installation guide for more detailed installation instructions.

Clone this repository

git clone https://github.com/TalkAI/facial-emotion-recognition-gluoncd facial-emotion-recognition-gluon

Download FER datasetfer2013.tar.gz from theFER Kaggle competition.

Note: You cannot download the dataset withwget. You will have to register on Kaggle, and then login to download the dataset.

Once downloaded:

• Extract the tar file -fer2013.tar.gz
• Copyfer2013.csv dataset tofacial-emotion-recognition-gluon/data directory.

We will now generateFER+ train/test/validation dataset from the downloadedFER data by executing the command below:

# In this step, we read the raw FER data, correct the labels using FER+ labels, and save as png images.# -d : path to "data" folder in this repository. It has folder for Train/Test/Validation data with corrected labels.# -fer : path to fer dataset that you have extracted.# -ferplus : path to fer2013new.csv file that comes with this repository in the data folder    python utils/prepare_data.py -d ./data -fer ./data/fer2013.csv -ferplus ./data/fer2013new.csv

Lastly, we will process theFER+ train/test/validation dataset

# This script reads the FER+ dataset (png images) we prepared in the previous step, applies the transformation suggested in the FER+ paper, and saves the processed images as NumPy binaries (npy files).# -d : path to data folder. This is where we have created data from the previous step.python utils/process_data.py -d ./data

Go overthis notebook that provides basic overview and intuitiion for various deep learning concepts and techniaues used in building and training the model.

Head over to theFER+ tutorial, to go over the process for building, training and deploying FER+ model. It is best to run as a live Jupyter Notebook, and you would need a GPU machine to complete training in a reasonable time.

Below are few areas of improvements and next steps for the advanced reader. Contributions back to the repository are welcomed!

• Hyper-parameter optimization - In this implementation, I have not optimized hyper-parameters (learning rate scheduler for SGD) for best possible result.
• Implement multi-gpu version of model training. This script provides single GPU implementation only. Time per epoch on single GPU is around 1 minute => approx 50 minutes for full model training (Model converges at around 50th epoch)

• Sandeep Krishnamurthy
• Saravana Kumar
• Hagay Lupesko

• Barsoum, Emad & Zhang, Cha & Canton Ferrer, Cristian & Zhang, Zhengyou (2016).Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution. In ACM International Conference on Multimodal Interaction (ICMI).
• CNTK implementation of FER+ paper -https://github.com/Microsoft/FERPlus
• FER demo built byhttps://github.com/codewithsk. (GitHub repo will be public soon)

• Apache MXNet (incubating) -http://mxnet.incubator.apache.org/
• Learn Deep Learning with Gluon -https://gluon.mxnet.io/
• Productionizing Deep Learning Models with MXNet Model Server -https://github.com/awslabs/mxnet-model-server