Repository files navigation

NeuralNetwork.NET is a .NET Standard 2.0 library that implements sequential and computation graph neural networks with customizable layers, built from scratch with C#.It provides simple APIs designed for quick prototyping to define and train models using stochastic gradient descent, as well as methods to save/load a network model and its metadata and more. The library also exposes CUDA-accelerated layers with more advanced features that leverage the GPU and the cuDNN toolkit to greatly increase the performances when training or using a neural network.

DISCLAIMER: this library is provided as is, and it's no longer being actively maintained. NeuralNetwork.NET was developed during a university course and it's not meant to be a replacement for other well known machine learning frameworks. If you're looking for a machine learning library for .NET to use in production, I recommend trying outML.NET or alternativelyTensorFlow.NET.

• Installing from NuGet
• Quick start
  • Supervised learning
  • GPU acceleration
  • Computation graphs
  • Library settings
  • Serialization and deserialization
  • Built-in datasets
• Requirements

To installNeuralNetwork.NET, run the following command in thePackage Manager Console

Install-Package NeuralNetwork.NET

More details availablehere.

TheNeuralNetwork.NET library exposes easy to use classes and methods to create a new neural network, prepare the datasets to use and train the network. These APIs are designed for rapid prototyping, and this section provides an overview of the required steps to get started.

The first step is to create a custom network structure. Here is an example with a sequential network (a stack of layers):

INeuralNetworknetwork=NetworkManager.NewSequential(TensorInfo.Image<Alpha8>(28,28),NetworkLayers.Convolutional((5,5),20,ActivationType.Identity),NetworkLayers.Pooling(ActivationType.LeakyReLU),NetworkLayers.Convolutional((3,3),40,ActivationType.Identity),NetworkLayers.Pooling(ActivationType.LeakyReLU),NetworkLayers.FullyConnected(125,ActivationType.LeakyReLU),NetworkLayers.FullyConnected(64,ActivationType.LeakyReLU),NetworkLayers.Softmax(10));

The next step is to prepare the datasets to use, through the APIs in theDatasetLoader class:

// A training dataset with a batch size of 100IEnumerable<(float[]x,float[]u)>data= ...// Your own dataset parsing routineITrainingDataset dataset=DatasetLoader.Training(data,100);// An optional test dataset with a callback to monitor the progressITestDatasettest=DatasetLoader.Test(..., p=>{Console.WriteLine($"Epoch{p.Iteration}, cost:{p.Cost}, accuracy:{p.Accuracy}");// Progress report});

Training a neural network is pretty straightforward - just use the methods in theNetworkManager class:

// Train the network using Adadelta and 0.5 dropout probabilityTrainingSessionResultresult=NetworkManager.TrainNetwork(network,// The network instance to traindataset,// The ITrainingDataset instanceTrainingAlgorithms.AdaDelta(),// The training algorithm to use60,// The expected number of training epochs to run0.5f,// Dropout probability    p=> ...,// Optional training epoch progress callbacknull,// Optional callback to monitor the training dataset accuracynull,// Optional validation datasettest,// Test datasettoken);// Cancellation token for the training

Note: theNetworkManager methods are also available as asynchronous APIs.

When running on a supported framework (.NET Framework, Xamarin or Mono), it is possible to use a different implementation of the available layers that leverages the cuDNN toolkit and parallelizes most of the work on the available CUDA-enabled GPU. To do that, just use the layers from theCuDnnNetworkLayers class when creating a network.

Some of the cuDNN-powered layers support additional options than the default layers. Here's an example:

// A cuDNN convolutional layer, with custom mode, padding and strideLayerFactoryconvolutional=CuDnnNetworkLayers.Convolutional(ConvolutionInfo.New(ConvolutionMode.CrossCorrelation,3,3,2,2),(7,7),20,ActivationType.ReLU);// An inception module, from the design of the GoogLeNet networkLayerFactoryinception=CuDnnNetworkLayers.Inception(InceptionInfo.New(10,// 1x1 convolution kernels20,10,// 1x1 + 3x3 convolution pipeline kernels20,10,// 1x1 + 5x5 convolution pipeline kernelsPoolingMode.AverageExcludingPadding,10));// Pooling mode and 1x1 convolution kernels

TheseLayerFactory instances can be used to create a new network just like in the CPU example.

NOTE: in order to use this feature, the CUDA and cuDNN toolkits must be installed on the current system, a CUDA-enabled nVidia GeForce/Quadro GPU must be available and theAlea NuGet package must be installed in the application using theNeuralNetwork.NET library as well. Additional info are availablehere.

Some complex network structures, like residual networks or inception modules , cannot be expressed as a simple sequential network structure: this is where computation graph networks come into play. Instead of forwarding the inputs through a linear stack of layers, a computation graph has a specific spatial structure that allows different nodes to be connected together. For example, it is possible to channel data through different parallel pipelines that are merged later on in the graph, or to have auxiliary classifiers that contribute to the gradient backpropagation during the training phase.

Computation graph networks are created using theNetworkManager.NewGraph API, here's an example:

INeuralNetworknetwork=NetworkManager.NewGraph(TensorInfo.Image<Rgb24>(32,32), root=>{varconv1=root.Layer(CuDnnNetworkLayers.Convolutional((5,5),20,ActivationType.Identity));varpool1=conv1.Layer(CuDnnNetworkLayers.Pooling(ActivationType.ReLU));varconv2=pool1.Pipeline(CuDnnNetworkLayers.Convolutional((1,1),20,ActivationType.ReLU),CuDnnNetworkLayers.Convolutional(ConvolutionInfo.Same(),(5,5),40,ActivationType.ReLU),CuDnnNetworkLayers.Convolutional((1,1),20,ActivationType.ReLU));varsum=conv2+pool1;varfc1=sum.Layer(CuDnnNetworkLayers.FullyConnected(250,ActivationType.LeCunTanh));varfc2=fc1.Layer(CuDnnNetworkLayers.FullyConnected(125,ActivationType.LeCunTanh));_=fc2.Layer(CuDnnNetworkLayers.Softmax(10));});

NeuralNetwork.NET provides various shared settings that are available through theNetworkSettings class.This class acts as a container to quickly check and modify any setting at any time, and these settings will influence the behavior of any existingINeuralNetwork instance and the library in general.

For example, it is possible to customize the criteria used by the networks to check their performance during training

NetworkSettings.AccuracyTester=AccuracyTesters.Argmax();// The default mode (mutually-exclusive classes)// Other testers are available tooNetworkSettings.AccuracyTester=AccuracyTesters.Threshold();// Useful for overlapping classesNetworkSettings.AccuracyTester=AccuracyTesters.Distance(0.2f);// Distance between results and expected outputs

When using CUDA-powered networks, sometimes the GPU in use might not be able to process the whole test or validation datasets in a single pass, which is the default behavior (these datasets are not divided into batches).To avoid memory issues, it is possible to modify this behavior:

NetworkSettings.MaximumBatchSize=400;// This will apply to any test or validation dataset

TheINeuralNetwork interface exposes aSave method that can be used to serialize any network at any given time.In order to get a new network instance from a saved file or stream, just use theNetworkLoader.TryLoad method.

As multiple layer types have different implementations across the available libraries, you can specify the layer providers to use when loading a saved network. For example, here's how to load a network using the cuDNN layers, when possible:

FileInfofile=newFileInfo(@"C:\...\MySavedNetwork.nnet");INeuralNetworknetwork=NetworkLoader.TryLoad(file,ExecutionModePreference.Cuda);

Note: theExecutionModePreference option indicates the desired type of layers to deserialize whenever possible. For example, usingExecutionModePreference.Cpu, the loaded network will only have CPU-powered layers, if supported.

There's also an additionalSaveMetadataAsJson method to export the metadata of anINeuralNetwork instance.

TheNeuralNetworkNET.Datasets namespace includes static classes to quickly load a popular dataset and get anIDataset instance ready to use with a new neural network. As an example, here's how to get the MNIST dataset:

ITrainingDatasettrainingData=awaitMnist.GetTrainingDatasetAsync(400);// Batches of 400 samplesITestDatasettestData=awaitMnist.GetTestDatasetAsync(p=> .../* Optional callback */);

Each API in this namespace also supports an optionalCancellationToken to stop the dataset loading, as the source data is downloaded from the internet and can take some time to be available, depending on the dataset being used.

TheNeuralNetwork.NET library requires .NET Standard 2.0 support, so it is available for applications targeting:

• .NET Framework >= 4.6.1
• .NET Core >= 2.0
• UWP (from SDK 10.0.16299)
• Mono >= 5.4
• Xamarin.iOS 10.14, Xamarin.Mac 3.8, Xamarin.Android 8.0

In addition to the frameworks above, you need an IDE with C# 7.3 support to compile the library on your PC.