Repository files navigation

Víctor Campos	Brendan Jou	Xavier Giro-i-Nieto

A joint collaboration between:

Barcelona Supercomputing Center (BSC)	Universitat Politecnica de Catalunya (UPC)	UPC Image Processing Group	Columbia University	Digital Video and Multimedia Lab (DVMM)

Visual multimedia have become an inseparable part of our digital social lives, and they often capture moments tied with deep affections. Automated visual sentiment analysis tools can provide a means of extracting the rich feelings and latent dispositions embedded in these media. In this work, we explore how Convolutional Neural Networks (CNNs), a now de facto computational machine learning tool particularly in the area of Computer Vision, can be specifically applied to the task of visual sentiment prediction. We accomplish this through fine-tuning experiments using a state-of-the-art CNN and via rigorous architecture analysis, we present several modifications that lead to accuracy improvements over prior art on a dataset of images from a popular social media platform. We additionally present visualizations of local patterns that the network learned to associate with image sentiment for insight into how visual positivity (or negativity) is perceived by the model.

Ourarticle can be found on ScienceDirect. Apreprint is publicly available on arXiv as well. You can also find it indexedon gitxiv.

Please cite with the following Bibtex code:

@article{campos2017from,  title={From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction},  author={Campos, Victor and Jou, Brendan and Giro-i-Nieto, Xavier},  journal={Image and Vision Computing},  year={2017}}

You may also want to refer to our publication with the more human-friendly APA style:

Campos, V., Jou, B., & Giro-i-Nieto, X. (2017, February).From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction. Image and Vision Computing.

The Twitter dataset used in our experiments was originally available fromthis URL from Rochester University. On September 2021, we noticed that it is no longer available, so we providethis mirror (57.4 MB).

The weights for the best CNN model can be downloaded fromhere (217 MB). These same weights, modified to fit the fully convolutional architecture used to generate the sentiment maps, can be downloaded fromhere (217 MB).

The deep network was developed overCaffe byBerkeley Vision and Learning Center (BVLC). You will need to followthese instructions to install Caffe.

We do not provide training code because we used Caffe's command line tool to train the models. Please see theframework's website for more details on how to download pre-trained models and fine-tune them on your data. Besides the trained models that can be used for inference, our repo providestext files with (image_id, label) tuples for all cross-validation splits in the paper. These can be used to train the model, but you will need to download the dataset from theproject site first.

We would like to especially thank Albert Gil and Josep Pujal from our technical support team at the Image Processing Group at UPC and Carlos Tripiana from the technical support team at the Barcelona Supercomputing Center.

Albert Gil	Josep Pujal	Carlos Tripiana

This work has been supported by thegrant SEV2015-0493 of the Severo Ochoa Program awarded by Spanish Government, project TIN2015-65316 by the Spanish Ministry of Science and Innovation contracts 2014-SGR-1051 by Generalitat de Catalunya
We gratefully acknowledge the support ofNVIDIA Corporation through the BSC/UPC NVIDIA GPU Center of Excellence.
The Image ProcessingGroup at the UPC is aSGR14 Consolidated Research Group recognized and sponsored by the Catalan Government (Generalitat de Catalunya) through itsAGAUR office.
This work has been developed in the framework of the projectBigGraph TEC2013-43935-R, funded by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF).

If you have any general doubt about our work or code which may be of interest for other researchers, please use thepublic issues section on this github repo.