Motivation#

Before we embark on our exciting journey of GANs and MNIST digits, let’s take astep back and discuss why you would want to use the C++ frontend instead of thePython one to begin with. We (the PyTorch team) created the C++ frontend toenable research in environments in which Python cannot be used, or is simply notthe right tool for the job. Examples for such environments include:

• Low Latency Systems: You may want to do reinforcement learning research ina pure C++ game engine with high frames-per-second and low latencyrequirements. Using a pure C++ library is a much better fit to such anenvironment than a Python library. Python may not be tractable at all becauseof the slowness of the Python interpreter.

• Highly Multithreaded Environments: Due to the Global Interpreter Lock(GIL), Python cannot run more than one system thread at a time.Multiprocessing is an alternative, but not as scalable and has significantshortcomings. C++ has no such constraints and threads are easy to use andcreate. Models requiring heavy parallelization, like those used inDeepNeuroevolution, can benefit fromthis.

• Existing C++ Codebases: You may be the owner of an existing C++application doing anything from serving web pages in a backend server torendering 3D graphics in photo editing software, and wish to integratemachine learning methods into your system. The C++ frontend allows you toremain in C++ and spare yourself the hassle of binding back and forth betweenPython and C++, while retaining much of the flexibility and intuitiveness ofthe traditional PyTorch (Python) experience.

The C++ frontend is not intended to compete with the Python frontend. It ismeant to complement it. We know researchers and engineers alike love PyTorch forits simplicity, flexibility and intuitive API. Our goal is to make sure you cantake advantage of these core design principles in every possible environment,including the ones described above. If one of these scenarios describes your usecase well, or if you are simply interested or curious, follow along as weexplore the C++ frontend in detail in the following paragraphs.

Writing a Basic Application#

Let’s begin by writing a minimal C++ application to verify that we’re on thesame page regarding our setup and build environment. First, you will need tograb a copy of theLibTorch distribution – our ready-built zip archive thatpackages all relevant headers, libraries and CMake build files required to usethe C++ frontend. The LibTorch distribution is available for download on thePyTorch website for Linux, MacOSand Windows. The rest of this tutorial will assume a basic Ubuntu Linuxenvironment, however you are free to follow along on MacOS or Windows too.

The first step is to download the LibTorch distribution locally, via the linkretrieved from the PyTorch website. For a vanilla Ubuntu Linux environment, thismeans running:

# If you need e.g. CUDA 9.0 support, please replace "cpu" with "cu90" in the URL below.wgethttps://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zipunziplibtorch-shared-with-deps-latest.zip

Next, let’s write a tiny C++ file calleddcgan.cpp that includestorch/torch.h and for now simply prints out a three by three identitymatrix:

#include<torch/torch.h>#include<iostream>intmain(){torch::Tensortensor=torch::eye(3);std::cout<<tensor<<std::endl;}

To build this tiny application as well as our full-fledged training script lateron we’ll use thisCMakeLists.txt file:

cmake_minimum_required(VERSION3.5FATAL_ERROR)project(dcgan)find_package(TorchREQUIRED)add_executable(dcgandcgan.cpp)target_link_libraries(dcgan"${TORCH_LIBRARIES}")set_property(TARGETdcganPROPERTYCXX_STANDARD17)

Make note of line 4 in the above CMake file:find_package(TorchREQUIRED).This instructs CMake to find the build configuration for the LibTorch library.In order for CMake to knowwhere to find these files, we must set theCMAKE_PREFIX_PATH when invokingcmake. Before we do this, let’s agree onthe following directory structure for ourdcgan application:

dcgan/CMakeLists.txtdcgan.cpp

Further, I will refer to the path to the unzipped LibTorch distribution as/path/to/libtorch. Note that thismust be an absolute path. Inparticular, settingCMAKE_PREFIX_PATH to something like../../libtorchwill break in unexpected ways. Instead, write$PWD/../../libtorch to get thecorresponding absolute path. Now, we are ready to build our application:

root@fa350df05ecf:/home#mkdirbuildroot@fa350df05ecf:/home#cdbuildroot@fa350df05ecf:/home/build#cmake-DCMAKE_PREFIX_PATH=/path/to/libtorch..--TheCcompileridentificationisGNU5.4.0--TheCXXcompileridentificationisGNU5.4.0--CheckforworkingCcompiler:/usr/bin/cc--CheckforworkingCcompiler:/usr/bin/cc--works--DetectingCcompilerABIinfo--DetectingCcompilerABIinfo-done--DetectingCcompilefeatures--DetectingCcompilefeatures-done--CheckforworkingCXXcompiler:/usr/bin/c++--CheckforworkingCXXcompiler:/usr/bin/c++--works--DetectingCXXcompilerABIinfo--DetectingCXXcompilerABIinfo-done--DetectingCXXcompilefeatures--DetectingCXXcompilefeatures-done--Lookingforpthread.h--Lookingforpthread.h-found--Lookingforpthread_create--Lookingforpthread_create-notfound--Lookingforpthread_createinpthreads--Lookingforpthread_createinpthreads-notfound--Lookingforpthread_createinpthread--Lookingforpthread_createinpthread-found--FoundThreads:TRUE--Foundtorch:/path/to/libtorch/lib/libtorch.so--Configuringdone--Generatingdone--Buildfileshavebeenwrittento:/home/buildroot@fa350df05ecf:/home/build#cmake--build.--configReleaseScanningdependenciesoftargetdcgan[50%]BuildingCXXobjectCMakeFiles/dcgan.dir/dcgan.cpp.o[100%]LinkingCXXexecutabledcgan[100%]Builttargetdcgan

Above, we first created abuild folder inside of ourdcgan directory,entered this folder, ran thecmake command to generate the necessary build(Make) files and finally compiled the project successfully by runningcmake--build.--configRelease. We are now all set to execute our minimal binaryand complete this section on basic project configuration:

root@fa350df05ecf:/home/build#./dcgan100010001[Variable[CPUFloatType]{3,3}]

Looks like an identity matrix to me!

Defining the Neural Network Models#

Now that we have our basic environment configured, we can dive into the muchmore interesting parts of this tutorial. First, we will discuss how to defineand interact with modules in the C++ frontend. We’ll begin with basic,small-scale example modules and then implement a full-fledged GAN using theextensive library of built-in modules provided by the C++ frontend.

importtorchclassNet(torch.nn.Module):def__init__(self,N,M):super(Net,self).__init__()self.W=torch.nn.Parameter(torch.randn(N,M))self.b=torch.nn.Parameter(torch.randn(M))defforward(self,input):returntorch.addmm(self.b,input,self.W)

#include<torch/torch.h>structNet:torch::nn::Module{Net(int64_tN,int64_tM){W=register_parameter("W",torch::randn({N,M}));b=register_parameter("b",torch::randn(M));}torch::Tensorforward(torch::Tensorinput){returntorch::addmm(b,input,W);}torch::TensorW,b;};

classNet(torch.nn.Module):def__init__(self,N,M):super(Net,self).__init__()# Registered as a submodule behind the scenesself.linear=torch.nn.Linear(N,M)self.another_bias=torch.nn.Parameter(torch.rand(M))defforward(self,input):returnself.linear(input)+self.another_bias

>>>net=Net(4,5)>>>print(list(net.parameters()))[Parameter containing:tensor([0.0808, 0.8613, 0.2017, 0.5206, 0.5353], requires_grad=True), Parameter containing:tensor([[-0.3740, -0.0976, -0.4786, -0.4928],        [-0.1434,  0.4713,  0.1735, -0.3293],        [-0.3467, -0.3858,  0.1980,  0.1986],        [-0.1975,  0.4278, -0.1831, -0.2709],        [ 0.3730,  0.4307,  0.3236, -0.0629]], requires_grad=True), Parameter containing:tensor([ 0.2038,  0.4638, -0.2023,  0.1230, -0.0516], requires_grad=True)]

structNet:torch::nn::Module{Net(int64_tN,int64_tM):linear(register_module("linear",torch::nn::Linear(N,M))){another_bias=register_parameter("b",torch::randn(M));}torch::Tensorforward(torch::Tensorinput){returnlinear(input)+another_bias;}torch::nn::Linearlinear;torch::Tensoranother_bias;};

intmain(){Netnet(4,5);for(constauto&p:net.parameters()){std::cout<<p<<std::endl;}}

root@fa350df05ecf:/home/build#./dcgan0.03451.4456-0.6313-0.3585-0.4008[Variable[CPUFloatType]{5}]-0.16470.28910.0527-0.03540.30840.20250.03430.1824-0.4630-0.28620.2500-0.04200.3679-0.1482-0.04600.19670.2132-0.19920.42570.0739[Variable[CPUFloatType]{5,4}]0.01*3.6861-10.1166-45.03337.9983-20.0705[Variable[CPUFloatType]{5}]

Netnet(4,5);for(constauto&pair:net.named_parameters()){std::cout<<pair.key()<<": "<<pair.value()<<std::endl;}

root@fa350df05ecf:/home/build#make&&./dcgan11:13:48Scanningdependenciesoftargetdcgan[50%]BuildingCXXobjectCMakeFiles/dcgan.dir/dcgan.cpp.o[100%]LinkingCXXexecutabledcgan[100%]Builttargetdcganb:-0.1863-0.8611-0.12281.32690.9858[Variable[CPUFloatType]{5}]linear.weight:0.03390.24840.2035-0.2103-0.0715-0.2975-0.4350-0.1878-0.36160.1050-0.49820.0335-0.16050.49630.4099-0.28830.1818-0.3447-0.1501-0.0215[Variable[CPUFloatType]{5,4}]linear.bias:-0.02500.04080.3756-0.2149-0.3636[Variable[CPUFloatType]{5}]

intmain(){Netnet(4,5);std::cout<<net.forward(torch::ones({2,4}))<<std::endl;}

root@fa350df05ecf:/home/build#./dcgan0.85591.15722.1069-0.12470.80600.85591.15722.1069-0.12470.8060[Variable[CPUFloatType]{2,5}]

structNet:torch::nn::Module{};voida(Netnet){}voidb(Net&net){}voidc(Net*net){}intmain(){Netnet;a(net);a(std::move(net));b(net);c(&net);}

structNet:torch::nn::Module{};voida(std::shared_ptr<Net>net){}intmain(){autonet=std::make_shared<Net>();a(net);}

structNet:torch::nn::Module{Net(int64_tN,int64_tM):linear(register_module("linear",torch::nn::Linear(N,M))){}torch::nn::Linearlinear;};

structLinearImpl:torch::nn::Module{LinearImpl(int64_tin,int64_tout);Tensorforward(constTensor&input);Tensorweight,bias;};TORCH_MODULE(Linear);

structNetImpl:torch::nn::Module{};TORCH_MODULE(Net);voida(Netnet){}intmain(){Netnet;a(net);}

structNet:torch::nn::Module{Net(int64_tN,int64_tM):linear(register_module("linear",torch::nn::Linear(N,M))){}torch::nn::Linearlinear;};

structNet:torch::nn::Module{Net(int64_tN,int64_tM){linear=register_module("linear",torch::nn::Linear(N,M));}torch::nn::Linearlinear{nullptr};// construct an empty holder};

structDCGANGeneratorImpl:nn::Module{DCGANGeneratorImpl(intkNoiseSize):conv1(nn::ConvTranspose2dOptions(kNoiseSize,256,4).bias(false)),batch_norm1(256),conv2(nn::ConvTranspose2dOptions(256,128,3).stride(2).padding(1).bias(false)),batch_norm2(128),conv3(nn::ConvTranspose2dOptions(128,64,4).stride(2).padding(1).bias(false)),batch_norm3(64),conv4(nn::ConvTranspose2dOptions(64,1,4).stride(2).padding(1).bias(false)){// register_module() is needed if we want to use the parameters() method later onregister_module("conv1",conv1);register_module("conv2",conv2);register_module("conv3",conv3);register_module("conv4",conv4);register_module("batch_norm1",batch_norm1);register_module("batch_norm2",batch_norm2);register_module("batch_norm3",batch_norm3);}torch::Tensorforward(torch::Tensorx){x=torch::relu(batch_norm1(conv1(x)));x=torch::relu(batch_norm2(conv2(x)));x=torch::relu(batch_norm3(conv3(x)));x=torch::tanh(conv4(x));returnx;}nn::ConvTranspose2dconv1,conv2,conv3,conv4;nn::BatchNorm2dbatch_norm1,batch_norm2,batch_norm3;};TORCH_MODULE(DCGANGenerator);DCGANGeneratorgenerator(kNoiseSize);

nn::Sequentialdiscriminator(// Layer 1nn::Conv2d(nn::Conv2dOptions(1,64,4).stride(2).padding(1).bias(false)),nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),// Layer 2nn::Conv2d(nn::Conv2dOptions(64,128,4).stride(2).padding(1).bias(false)),nn::BatchNorm2d(128),nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),// Layer 3nn::Conv2d(nn::Conv2dOptions(128,256,4).stride(2).padding(1).bias(false)),nn::BatchNorm2d(256),nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),// Layer 4nn::Conv2d(nn::Conv2dOptions(256,1,3).stride(1).padding(0).bias(false)),nn::Sigmoid());

Loading Data#

Now that we have defined the generator and discriminator model, we need somedata we can train these models with. The C++ frontend, like the Python one,comes with a powerful parallel data loader. This data loader can read batches ofdata from a dataset (which you can define yourself) and provides manyconfiguration knobs.

The data loader is part of the C++ frontend’sdata api, contained in thetorch::data:: namespace. This API consists of a few different components:

• The data loader class,

• An API for defining datasets,

• An API for definingtransforms, which can be applied to datasets,

• An API for definingsamplers, which produce the indices with which datasets are indexed,

• A library of existing datasets, transforms and samplers.

For this tutorial, we can use theMNIST dataset that comes with the C++frontend. Let’s instantiate atorch::data::datasets::MNIST for this, andapply two transformations: First, we normalize the images so that they are inthe range of-1 to+1 (from an original range of0 to1).Second, we apply theStackcollation, which takes a batch of tensors andstacks them into a single tensor along the first dimension:

autodataset=torch::data::datasets::MNIST("./mnist").map(torch::data::transforms::Normalize<>(0.5,0.5)).map(torch::data::transforms::Stack<>());

Note that the MNIST dataset should be located in the./mnist directoryrelative to wherever you execute the training binary from. You can usethisscriptto download the MNIST dataset.

Next, we create a data loader and pass it this dataset. To make a new dataloader, we usetorch::data::make_data_loader, which returns astd::unique_ptr of the correct type (which depends on the type of thedataset, the type of the sampler and some other implementation details):

autodata_loader=torch::data::make_data_loader(std::move(dataset));

The data loader does come with a lot of options. You can inspect the full sethere.For example, to speed up the data loading, we can increase the number ofworkers. The default number is zero, which means the main thread will be used.If we setworkers to2, two threads will be spawned that load dataconcurrently. We should also increase the batch size from its default of1to something more reasonable, like64 (the value ofkBatchSize). Solet’s create aDataLoaderOptions object and set the appropriate properties:

autodata_loader=torch::data::make_data_loader(std::move(dataset),torch::data::DataLoaderOptions().batch_size(kBatchSize).workers(2));

We can now write a loop to load batches of data, which we’ll only print to theconsole for now:

for(torch::data::Example<>&batch:*data_loader){std::cout<<"Batch size: "<<batch.data.size(0)<<" | Labels: ";for(int64_ti=0;i<batch.data.size(0);++i){std::cout<<batch.target[i].item<int64_t>()<<" ";}std::cout<<std::endl;}

The type returned by the data loader in this case is atorch::data::Example.This type is a simple struct with adata field for the data and atargetfield for the label. Because we applied theStack collation earlier, thedata loader returns only a single such example. If we had not applied thecollation, the data loader would yieldstd::vector<torch::data::Example<>>instead, with one element per example in the batch.

If you rebuild and run this code, you should see something like this:

root@fa350df05ecf:/home/build#makeScanningdependenciesoftargetdcgan[50%]BuildingCXXobjectCMakeFiles/dcgan.dir/dcgan.cpp.o[100%]LinkingCXXexecutabledcgan[100%]Builttargetdcganroot@fa350df05ecf:/home/build#make[100%]Builttargetdcganroot@fa350df05ecf:/home/build#./dcganBatchsize:64|Labels:5267216701623691840653304666408606924028633292014234829935800799Batchsize:64|Labels:2247128869022936138044888926471509754354128071961653441232350162Batchsize:64|Labels:4542148383615436225131508215324459728920674383588305808785561780Batchsize:64|Labels:3371416103640254042819651632892387459608300648254183780089672147Batchsize:64|Labels:3055983989595041277200548776107930632627633405889192194492462940Batchsize:64|Labels:9675359086678219881182071416751774032906634481286920312856485862Batchsize:64|Labels:9303651860199161774447886782604682539840993705824562825371918227Batchsize:64|Labels:9192726086877486116857913205173161086081054938584801262427737453Batchsize:64|Labels:8831864295802866709838716627745521795491031939885375368942012547Batchsize:64|Labels:9270844275006205959889357547305765716287632656127700590091783294Batchsize:64|Labels:7657752249948748945712698512367811398795085187265120974090460086...

Which means we are successfully able to load data from the MNIST dataset.

Writing the Training Loop#

Let’s now finish the algorithmic part of our example and implement the delicatedance between the generator and discriminator. First, we’ll create twooptimizers, one for the generator and one for the discriminator. The optimizerswe use implement theAdam algorithm:

torch::optim::Adamgenerator_optimizer(generator->parameters(),torch::optim::AdamOptions(2e-4).betas(std::make_tuple(0.5,0.5)));torch::optim::Adamdiscriminator_optimizer(discriminator->parameters(),torch::optim::AdamOptions(5e-4).betas(std::make_tuple(0.5,0.5)));

Next, we need to update our training loop. We’ll add an outer loop to exhaustthe data loader every epoch and then write the GAN training code:

for(int64_tepoch=1;epoch<=kNumberOfEpochs;++epoch){int64_tbatch_index=0;for(torch::data::Example<>&batch:*data_loader){// Train discriminator with real images.discriminator->zero_grad();torch::Tensorreal_images=batch.data;torch::Tensorreal_labels=torch::empty(batch.data.size(0)).uniform_(0.8,1.0);torch::Tensorreal_output=discriminator->forward(real_images).reshape(real_labels.sizes());torch::Tensord_loss_real=torch::binary_cross_entropy(real_output,real_labels);d_loss_real.backward();// Train discriminator with fake images.torch::Tensornoise=torch::randn({batch.data.size(0),kNoiseSize,1,1});torch::Tensorfake_images=generator->forward(noise);torch::Tensorfake_labels=torch::zeros(batch.data.size(0));torch::Tensorfake_output=discriminator->forward(fake_images.detach()).reshape(fake_labels.sizes());torch::Tensord_loss_fake=torch::binary_cross_entropy(fake_output,fake_labels);d_loss_fake.backward();torch::Tensord_loss=d_loss_real+d_loss_fake;discriminator_optimizer.step();// Train generator.generator->zero_grad();fake_labels.fill_(1);fake_output=discriminator->forward(fake_images).reshape(fake_labels.sizes());torch::Tensorg_loss=torch::binary_cross_entropy(fake_output,fake_labels);g_loss.backward();generator_optimizer.step();std::printf("\r[%2ld/%2ld][%3ld/%3ld] D_loss: %.4f | G_loss: %.4f",epoch,kNumberOfEpochs,++batch_index,batches_per_epoch,d_loss.item<float>(),g_loss.item<float>());}}

Above, we first evaluate the discriminator on real images, for which it shouldassign a high probability. For this, we usetorch::empty(batch.data.size(0)).uniform_(0.8,1.0) as the targetprobabilities.

Before evaluating the discriminator, we zero out the gradients of itsparameters. After computing the loss, we back-propagate it through the network bycallingd_loss.backward() to compute new gradients. We repeat this spiel forthe fake images. Instead of using images from the dataset, we let the generatorcreate fake images for this by feeding it a batch of random noise. We thenforward those fake images to the discriminator. This time, we want thediscriminator to emit low probabilities, ideally all zeros. Once we havecomputed the discriminator loss for both the batch of real and the batch of fakeimages, we can progress the discriminator’s optimizer by one step in order toupdate its parameters.

To train the generator, we again first zero its gradients, and then re-evaluatethe discriminator on the fake images. However, this time we want thediscriminator to assign probabilities very close to one, which would indicatethat the generator can produce images that fool the discriminator into thinkingthey are actually real (from the dataset). For this, we fill thefake_labelstensor with all ones. We finally step the generator’s optimizer to also updateits parameters.

We should now be ready to train our model on the CPU. We don’t have any code yetto capture state or sample outputs, but we’ll add this in just a moment. Fornow, let’s just observe that our model is doingsomething – we’ll laterverify based on the generated images whether this something is meaningful.Re-building and running should print something like:

root@3c0711f20896:/home/build#make&&./dcganScanningdependenciesoftargetdcgan[50%]BuildingCXXobjectCMakeFiles/dcgan.dir/dcgan.cpp.o[100%]LinkingCXXexecutabledcgan[100%]Builttargetdcga[1/10][100/938]D_loss:0.6876|G_loss:4.1304[1/10][200/938]D_loss:0.3776|G_loss:4.3101[1/10][300/938]D_loss:0.3652|G_loss:4.6626[1/10][400/938]D_loss:0.8057|G_loss:2.2795[1/10][500/938]D_loss:0.3531|G_loss:4.4452[1/10][600/938]D_loss:0.3501|G_loss:5.0811[1/10][700/938]D_loss:0.3581|G_loss:4.5623[1/10][800/938]D_loss:0.6423|G_loss:1.7385[1/10][900/938]D_loss:0.3592|G_loss:4.7333[2/10][100/938]D_loss:0.4660|G_loss:2.5242[2/10][200/938]D_loss:0.6364|G_loss:2.0886[2/10][300/938]D_loss:0.3717|G_loss:3.8103[2/10][400/938]D_loss:1.0201|G_loss:1.3544[2/10][500/938]D_loss:0.4522|G_loss:2.6545...

Moving to the GPU#

While our current script can run just fine on the CPU, we all know convolutionsare a lot faster on GPU. Let’s quickly discuss how we can move our training ontothe GPU. We’ll need to do two things for this: pass a GPU device specificationto tensors we allocate ourselves, and explicitly copy any other tensors onto theGPU via theto() method all tensors and modules in the C++ frontend have.The simplest way to achieve both is to create an instance oftorch::Deviceat the top level of our training script, and then pass that device to tensorfactory functions liketorch::zeros as well as theto() method. We canstart by doing this with a CPU device:

// Place this somewhere at the top of your training script.torch::Devicedevice(torch::kCPU);

New tensor allocations like

torch::Tensorfake_labels=torch::zeros(batch.data.size(0));

should be updated to take thedevice as the last argument:

torch::Tensorfake_labels=torch::zeros(batch.data.size(0),device);

For tensors whose creation is not in our hands, like those coming from the MNISTdataset, we must insert explicitto() calls. This means

torch::Tensorreal_images=batch.data;

becomes

torch::Tensorreal_images=batch.data.to(device);

and also our model parameters should be moved to the correct device:

generator->to(device);discriminator->to(device);

At this point, we’ve just made our previous CPU-residing code more explicit.However, it is now also very easy to change the device to a CUDA device:

torch::Devicedevice(torch::kCUDA)

And now all tensors will live on the GPU, calling into fast CUDA kernels for alloperations, without us having to change any downstream code. If we wanted tospecify a particular device index, it could be passed as the second argument totheDevice constructor. If we wanted different tensors to live on differentdevices, we could pass separate device instances (for example one on CUDA device0 and the other on CUDA device 1). We can even do this configurationdynamically, which is often useful to make our training scripts more portable:

torch::Devicedevice=torch::kCPU;if(torch::cuda::is_available()){std::cout<<"CUDA is available! Training on GPU."<<std::endl;device=torch::kCUDA;}

or even

torch::Devicedevice(torch::cuda::is_available()?torch::kCUDA:torch::kCPU);

Checkpointing and Recovering the Training State#

The last augmentation we should make to our training script is to periodicallysave the state of our model parameters, the state of our optimizers as well as afew generated image samples. If our computer were to crash in the middle of thetraining procedure, the first two will allow us to restore the training state.For long-lasting training sessions, this is absolutely essential. Fortunately,the C++ frontend provides an API to serialize and deserialize both model andoptimizer state, as well as individual tensors.

The core API for this istorch::save(thing,filename) andtorch::load(thing,filename), wherething could be atorch::nn::Module subclass or an optimizer instance like theAdam objectwe have in our training script. Let’s update our training loop to checkpoint themodel and optimizer state at a certain interval:

if(batch_index%kCheckpointEvery==0){// Checkpoint the model and optimizer state.torch::save(generator,"generator-checkpoint.pt");torch::save(generator_optimizer,"generator-optimizer-checkpoint.pt");torch::save(discriminator,"discriminator-checkpoint.pt");torch::save(discriminator_optimizer,"discriminator-optimizer-checkpoint.pt");// Sample the generator and save the images.torch::Tensorsamples=generator->forward(torch::randn({8,kNoiseSize,1,1},device));torch::save((samples+1.0)/2.0,torch::str("dcgan-sample-",checkpoint_counter,".pt"));std::cout<<"\n-> checkpoint "<<++checkpoint_counter<<'\n';}

wherekCheckpointEvery is an integer set to something like100 tocheckpoint every100 batches, andcheckpoint_counter is a counter bumpedevery time we make a checkpoint.

To restore the training state, you can add lines like these after all models andoptimizers are created, but before the training loop:

torch::optim::Adamgenerator_optimizer(generator->parameters(),torch::optim::AdamOptions(2e-4).beta1(0.5));torch::optim::Adamdiscriminator_optimizer(discriminator->parameters(),torch::optim::AdamOptions(2e-4).beta1(0.5));if(kRestoreFromCheckpoint){torch::load(generator,"generator-checkpoint.pt");torch::load(generator_optimizer,"generator-optimizer-checkpoint.pt");torch::load(discriminator,"discriminator-checkpoint.pt");torch::load(discriminator_optimizer,"discriminator-optimizer-checkpoint.pt");}int64_tcheckpoint_counter=0;for(int64_tepoch=1;epoch<=kNumberOfEpochs;++epoch){int64_tbatch_index=0;for(torch::data::Example<>&batch:*data_loader){

Inspecting Generated Images#

Our training script is now complete. We are ready to train our GAN, whether onCPU or GPU. To inspect the intermediary output of our training procedure, forwhich we added code to periodically save image samples to the"dcgan-sample-xxx.pt" file, we can write a tiny Python script to load thetensors and display them with matplotlib:

importargparseimportmatplotlib.pyplotaspltimporttorchparser=argparse.ArgumentParser()parser.add_argument("-i","--sample-file",required=True)parser.add_argument("-o","--out-file",default="out.png")parser.add_argument("-d","--dimension",type=int,default=3)options=parser.parse_args()module=torch.jit.load(options.sample_file)images=list(module.parameters())[0]forindexinrange(options.dimension*options.dimension):image=images[index].detach().cpu().reshape(28,28).mul(255).to(torch.uint8)array=image.numpy()axis=plt.subplot(options.dimension,options.dimension,1+index)plt.imshow(array,cmap="gray")axis.get_xaxis().set_visible(False)axis.get_yaxis().set_visible(False)plt.savefig(options.out_file)print("Saved ",options.out_file)

Let’s now train our model for around 30 epochs:

root@3c0711f20896:/home/build#make&&./dcgan10:17:57Scanningdependenciesoftargetdcgan[50%]BuildingCXXobjectCMakeFiles/dcgan.dir/dcgan.cpp.o[100%]LinkingCXXexecutabledcgan[100%]BuilttargetdcganCUDAisavailable!TrainingonGPU.[1/30][200/938]D_loss:0.4953|G_loss:4.0195->checkpoint1[1/30][400/938]D_loss:0.3610|G_loss:4.8148->checkpoint2[1/30][600/938]D_loss:0.4072|G_loss:4.36760->checkpoint3[1/30][800/938]D_loss:0.4444|G_loss:4.0250->checkpoint4[2/30][200/938]D_loss:0.3761|G_loss:3.8790->checkpoint5[2/30][400/938]D_loss:0.3977|G_loss:3.3315...->checkpoint120[30/30][938/938]D_loss:0.3610|G_loss:3.8084

And display the images in a plot:

root@3c0711f20896:/home/build#pythondisplay.py-idcgan-sample-100.ptSavedout.png

Which should look something like this:

Digits! Hooray! Now the ball is in your court: can you improve the model to makethe digits look even better?

Conclusion#

This tutorial has hopefully given you a digestible digest of the PyTorch C++frontend. A machine learning library like PyTorch by necessity has a very broadand extensive API. As such, there are many concepts we did not have time orspace to discuss here. However, I encourage you to try out the API, and consultour documentation and in particular theLibrary API section whenyou get stuck. Also, remember that you can expect the C++ frontend to follow thedesign and semantics of the Python frontend whenever we could make thispossible, so you can leverage this fact to increase your learning rate.

As always, if you run into any problems or have questions, you can use ourforum orGitHub issues to get in touch.