The focus of this chapter is to provide solutions to common implementation problems for FNN and other network topologies. The techniques discussed in this chapter also apply to the following chapters.

FNNs are networks where the information only moves in one direction and does not cycle (as we will see inChapter 4,Recurrent Neural Networks). FNNs are mainly used for supervised learning where the data is not sequential or time-dependent, for example for general classification and regression tasks. We will start by introducing a perceptron and we will show how to implement a perceptron with NumPy. A perceptron demonstrates the mechanics of a single unit. Next, we will increase the complexity by increasing the number of units and introduce single-layer and multi-layer neural networks. The high number of units, in combination with a high number of layers, gives the depth of the architecture and is responsible for the name deep learning. 

First, we need to understand the basics of neural networks. A neural consists of one or multiple layers ofneurons, named after the neurons in human brains. We will demonstrate the mechanics of a single neuron by implementing a perceptron. In a perceptron, a single unit (neuron) performs all the computations. Later, we will scale the number of units to create deep neural networks:

Figure 2.1: Perceptron

A can have multiple inputs. On these inputs, the unit performs some computations and outputs a single value, for example a binary value to classify two classes. The computations performed by the unit are a simple matrix multiplication of the input and the weights. The resulting values are summed up and a bias is added:

These computations can easily be scaled to high dimensional input. Anactivation function (φ) determines the final output of the in theforward pass:

The weights and bias are initialized. After eachepoch (iteration over the training data), the...

Now we can move on toneural networks. We will start by the simplest form of a neural network: a single-layer neural network. The difference from a perceptron is that the computations are done by multiple units (neurons), hence a network. As you may expect, adding more units will increase the number of problems that can be solved. The units perform their computations separately and are in a layer; we call this layer thehidden layer. Therefore, we call the units in this layer the hidden units. For now, we will only consider a single hidden layer. The output layer performs as a perceptron. This time, as input we have the hidden units in the hidden layer instead of the input variables:

Figure 2.4: Single-layer neural network with two input variables, n hidden units, and a single output unit

In our implementation of the perceptron, we've used a unit step function to determine the class. In the next recipe, we will use a non-linear activation function...

What we've created in the recipe is actually the simplest form of an FNN: a neural network where the information flows only in one direction. For our next recipe, we will extend the number of hidden layers from one to multiple layers. Adding additional layers increases the power of a network to learn complex non-linear patterns. 

Figure 2.7: Two-layer neural network withi input variables, n hidden units, and m hidden units respectively, and a single output unit

As you can see inFigure 2-7, by adding an additional layer the number of connections (weights), also called trainable parameters, increases exponentially. In the next recipe, we will create a network with two hidden layers to predict wine quality. This is a regression task, so we will be using a linear activation for the output layer. For the hidden layers, we use ReLU activation functions. This recipe uses the Keras framework to implement the feed-forward network.

If we only use linear activation functions, a neural network would represent a large collection of linear combinations. However, the power of neural networks lies in their ability to model complex nonlinear behavior. We briefly introduced the non-linear activation functions sigmoid and ReLU in the previous recipes, and there are many more popular nonlinear functions, such asELU, Leaky ReLU,TanH, andMaxout.

There is no rule as to activation works best for the units. Deep learning is a new field and most results are obtained by trial and error instead of mathematical proofs. For the output unit, we use a single output unit and a linear activation for regression tasks. For classification tasks with n classes, we usen output nodes and a softmax activation function. The softmax function forces the network to output probabilities between0 and1 for mutually exclusive classes and the probabilities sum up to1. For binary classification, we can...

The most commonly used layers in neural networks are fully-connected layers. In fully-connected layers, the units in two successive layers are all  connected. However, the units within a layer don't share any connections. As stated before, the connections between the layers are also called trainable parameters. The weights of these connections are trained by the network. The more connections, the more parameters and the more complex patterns can be modeled. Most state-of-the-art models have 100+ million parameters. However, a deep neural network with many layers and units takes more time to train. Also, with extremely deep models the time to infer predictions takes significantly longer (which can be problematic in a real-time environment). In the following chapters, we will introduce other popular layer types that are specific to their network types. 

Picking the correct number of hidden layers and hidden units can be important. When using too...

For autoencoders, we use a network architecture, as shown in the following figure. In the first couple of layers, we decrease the number of hidden units. Halfway, we start increasing the number of hidden units again until the number of hidden units is the same as the number of input variables. The middle hidden layer can be seen as an encoded variant of the inputs, where the output determines the quality of the encoded variant:

Figure 2.13: Autoencoder network with three hidden layers, with m < n

In the next recipe, we will implement an in Keras to decodeStreet View House Numbers (SVHN) from 32 x 32 images to 32 floating numbers. We can determine the quality of the encoder by decoding back to 32 x 32 and comparing the images.

import numpy as npfrom matplotlib import pyplot as pltimport scipy.iofrom keras.models import Sequentialfrom keras.layers.core import Densefrom keras.optimizers...

While training a neural network for a learning problem, the objective of the network is to minimize the loss function. The loss function — also known as error, cost function, or opimization function–compares the prediction with the ground truth during the forward pass. The output of this loss function is used to optimize the weights during the backward pass. Therefore, the loss function is crucial in training the network. By setting the correct loss function, we force the network to optimize towards the desired predictions. For example, for imbalanced datasets we need a different loss function.

In the previous recipes, we've usedmean squared error (MSE) and categorical entropy as loss functions. There are also other popular loss functions, and another option is to create a custom loss function. A custom loss function gives the ability to optimize to the desired output. This will be important when we will implementGenerative Adversarial Networks (GANs). In the...

The most popular and well optimizer isStochastic Gradient Descent (SGD). This technique is widely used in other machine learning models as well. SGD is a to find minima or maxima by iteration. There are many popular variants of SGD that try to speed up convergence and less tuning by using an adaptive learning rate. Thefollowing table is an overview of the most commonly used optimizers in deep learning:

Overfitting on the data is one of the biggest of machine learning. There are many machine learning algorithms that are able to train on the training data by remembering all cases. In this scenario, the algorithm might not be able to generalize and make a correct prediction on new data. This is an especially big threat for deep learning, where neural networks have large numbers of trainable parameters. Therefore, it is extremely important to create a representative validation set. 

Most of the techniques used to prevent overfitting can be placed underregularization. Regularization include...

Another popular method for regularization is dropout. A forces a neural network to learn multiple independent representations by randomly removing connections between neurons in the learning phase. For example, when using a dropout of 0.5, the network has to see each example twice before the connection is learned. Therefore, a network with dropout can be seen as an ensemble of networks. 

In the following recipe, we will improve a model that clearly overfits the training data by adding dropouts.

import numpy as np import pandas as pdfrom sklearn.model_selection import train_test_splitfrom keras.models import Sequentialfrom keras.layers import Dense, Dropoutfrom keras.wrappers.scikit_learn import KerasRegressorfrom sklearn.model_selection import cross_val_scorefrom sklearn.model_selection import KFoldfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import Pipelineimport numpy as np...