Repository files navigation

Code for reproducing some key results of our NIPS 2014 paper on semi-supervised learning (SSL) with deep generative models.

D.P. Kingma, D.J. Rezende, S. Mohamed, M. Welling
Semi-Supervised Learning with Deep Generative Models
Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal
http://arxiv.org/abs/1406.5298

Please cite this paper when using this code for your research.

Warning: This code is far from fully commented.

For questions and bug reports, please send me an e-mail atdpkingma[at]gmail.com.

1. Make sure that recent versions installed of:

   • Python (version 2.7 or higher)
   • Numpy (e.g.pip install numpy)
   • Theano (e.g.pip install Theano)

2. SetfloatX = float32 in the[global] section of Theano config (usually~/.theanorc). Alternatively you could prependTHEANO_FLAGS=floatX=float32 to the python commands below.

3. Clone this repository, e.g.:

git clone https://github.com/dpkingma/nips14-ssl.git

4. Set an environment variableML_DATA_PATH that points to subdirectorydata/. For example, if you checked out this repo to your home directory:

export ML_DATA_PATH="$HOME/nips14-ssl/data"

To generate movies of flying through latent-space of the M2 model, run:

python run_flying.py [dataset] 1 output.mkv

wheredataset is 'mnist' or 'svhn', andtarget_filename is the filename to save the movie file to. NOTE: This script requires ffmpeg to be installed.

Run:

python run_analogies.py [dataset] 1

To train model M1 (a standard Variational Auto-Encoder / DLGM with sperical Gaussian latent space):

python run_gpulearn_z_x.py [dataset]

The M1 model does not incorporate class label, but is used in the paper's experiments for feature extration.

To run the semi-supervised learning experiments with model M1+M2:

python run_2layer_ssl.py [n_labels] [seed]

wheren_labels is the number of labels, andseed is the random seed for Numpy. To reproduce the experimental results in the paper, the number of labels should be in (100,600,1000,3000). The random seed can be any integer. Each experiment will run for 3000 epochs; since this code is not GPU-optimized, running many epochs might take a few days to complete. However, it is often not necessary to run the the algorithm for so many epochs to produce good results.

For training a generative model with all labels:

python run_gpulearn_yz_x.py [dataset]

wheredataset is 'mnist', 'svhn', 'norb' or 'norb_reshuffled'.

For evaluating the test-set classification error using already trained generative models of MNIST and SVHN:

python run_sl.py [dataset]

This iteratively builds, for each test-set image, an importance-sampled estimate of the posterior probability distribution over the class labels. This is an expensive procedure, but may be speed up by using fitting an inference model to the posterior distribution of class labels (which wasn't done in this case).