\subsection{Deep Learning \index{Deep Learning}}

\label{sec:dl}

The Deep Learning module in TMVA provides the support to build several deep

learning architectures such as Deep Neural Networks (DNN), Convolutional networks (CNN) and

Recurrent networks (RNN).

The purpose of this module is to provide

optimized implementations of these networks

that can be efficiently trained on modern multi-core and GPU

hardware architectures.

For this, the implementation provides two

backends for training and evaluating deep learning models.

The {\tt CPU} backend uses multithreading to perform

the training in parallel on multi-core CPU architectures and it can make use of

efficient multithreaded BLAS implementations such as OpenBLAS and the Intel TBB library.

The {\tt GPU} backend can be used to perform the training on

CUDA-capable GPU architectures. In the case of Convolutional and Recurrent network models

the implementation uses the low level CuDNN library of CUDA for optimal performances.

\paragraph{Building the CPU and GPU Backends}

To be able to use the multithreaded CPU backend, an implementation of the BLAS

library as well as the Intel TBB library must be available on the

system and detected by CMake. Note that the BLAS implementation must be

multithreaded in order to exploit the multithreading capabilities of

the underlying hardware.

The Intel TBB library is used by the ROOT multi-threading libraries and it can be also part of

ROOT build. Note that the \code{IMT} CMake flag must be set

to enable multithreading in ROOT.

If a BLAS implementation is not available, but ROOT is built with support for

the MathMore library that is based on Gnu Scientific Library (GSL), the CBLAS implementation provided by GSL will be used .

If neither BLAS or CBLAS are found, then the native ROOT matrix package will be used for

the matrix multiplications needed to evaluate and train deep learning models.

If ROOT has been built without IMT (multi-threading support) the execution will be serial instead of parallel.

For the GPU backend, a CUDA installation is required on the system and

must be successfully detected by CMake. This requires the user to set the

\code{CUDA} CMake flag when installing Root.

Furthermore, for using CNN and RNN networks on GPU the CUDA cuDNN library

must be installed and detected by CMAKE (\code{CUDNN} CMAKE flag).

If the user tries to use a backend that is not installed (e.g GPU), the training

of the DNN method will be performed using the alternative one (e.g. CPU).

\subsubsection{Deep Neural Networks \index{Deep Neural Networks}}

\label {sec:dnn}

A \textit{deep neural network} (DNN) is an artificial neural network

(see Section~\ref{sec:ann}) with several hidden layers and a large

number of neurons in each layer. Recent developments in machine

learning have shown that these networks are capable of learning

complex, non-linear relations when trained on a sufficiently large

amount of training data. For an article describing the application of

DNNs to a classification problem in particle physics see the article

by Baldi, Sadowski and Whiteson \cite{higgs_dnn}.

The deep neural network implementation provides

an optimized implementation of feed-forward multilayer perceptrons

that can be efficiently trained on modern multi-core and GPU

architectures.

Note that previous ROOT/TMVA versions (version $<$ 6.22) provide for Deep Neural Networks a

method {\tt TMVA::kDNN}, implementing only fully connected layers.

Since ROOT version 6.18 it is recommended to use the new {\tt TMVA::kDL} method, which supports

not only dense layer as in the previous implementation, but also new type of layers as

convolutional and recurrent, allowing to build convolutional and recurrent networks, as

we will see in the next paragraphs.

\subsubsection{Training of Deep Learning Models}

\label{sec:dnn:update}

As common for deep neural networks, this implementation uses several

methods to train the network.

As example we report here the formula for the \textit{stochastic batch gradient descend} (SGD).

This means that in each training step the weights $W^k_{i,j}$ and bias

terms $\theta_{i}^k$ of a given layer $k$ are updated using

\begin{align}

\end{align}

Here $J(\mathbf{x}_b,\mathbf{y}_b)$ is the value of the loss function corresponding to

the randomly chosen input batch $\mathbf{x}_b$ and expected output

$\mathbf{y}_b$. If regularization is applied, the loss may as well contain

contributions from the weight terms $W^k_{i,j}$ of each layer. This implementation also supports training with momentum. In this case training updates take the form:

\begin{align}

\end{align}

Note that for $p = 0$ the standard stochastic batch gradient descent method is

obtained.

Other optimizer algorithms such as ADAM, which is the default one since ROOT version 6.18, ADAGRAD, RMSPROP and ADADELTA

are available as optmization algorithms, through the {\it Optimizer} option of the training strategy.

For a detailed description of the various method see ~\cite{dl_optimizers}.

The training batch $(\mathbf{x}_b,\mathbf{y}_b)$ is chosen randomly

and without replacement from the set of training samples. The number

of training steps necessary to traverse the whole training set one time is

referred to as an \textit{epoch}.

The training of the deep neural network is performed in one or

multiple training phases. A training phase ends when the test error

did not decrease for a user-specified number of epochs.

\paragraph{Validation and Evaluation Set}

\label{sec:dnn_test_and_evaluation_set}

The user should note that this implementation splits the TMVA training set in a sub-training set used to compute the gradient and update the weights and a validation set used to compute the error and

to check for convergence. To properly evaluate the performance, the

model is later evaluated on a completely separate data set, the TMVA test set.

\subsubsection{Booking of the DL Method}

The booking of the DL method is similar to any other method in TMVA. The method

specific options are passed in the form of an option string \code{<options>} to

the method.

\begin{codeexample}

\begin{tmvacode}

factory->BookMethod(dataloader, TMVA::Types::kDL, "DNN", <options>);

\end{tmvacode}

\caption[.]{\codeexampleCaptionSize Booking of a deep neural network:

The first argument is a \code{TMVA::DataLoader} object, which holds training and test

data. The second argument is the predefined enumerator object that represents

the deep neural network implementation. The third argument is a string holding

a user defined name for the method. The fourth argument is the option string

holding the options for the method. See Sec.~\ref{sec:usingtmva:booking}

for more information on the booking.}

\label{example:DNN_booking}

\end{codeexample}

Through the option string, several configuration parameters can be specified.

The complete set of options that can be set in the options string are summarized in

Table~\ref{opt:mva:dnn:options}.

Some of the option such as the layout, define the models and the layers of the newtork and will be

described in greater details later, when describing the different deep leanring models that can be built.

The \code{Architecture} parameter is used to specify the backend used for the

training. The multi-core CPU and GPU backends must be enabled during the installation of Root

(CMAKE flgas {\code{tmva-cpu} and \code{tmva-gpu}).

The method used to initialize the weight matrices before beginning the

training is specified using the \code{WeightInitialization}

argument. Possible values are \code{XAVIER}, which initializes the

weights with zero-mean Gaussian distributed random values, and

\code{XAVIERUNIFORM}, which initializes the weights with uniformly

distributed values.

\begin{option}[h]

\input optiontables/MVA__DNN_1.tex

\caption[.]{\optionCaptionSize

Configuration options reference for MVA method: {\em DL}.

}

\label{opt:mva:dnn:options}

\end{option}

\subsubsection{Network Layout}

\label{sec:dnn:layout}

The structure of the deep learning model is specified by the

\code{Layout} parameter of the option string. This string specifies

the type of layer (e.g. dense, convolutional, recurrent ),

the activation function and some specific layer parameters such as the number of neurons of each layer of the

network or the filter sizes for the convolutional layers. Different layers are separated by '\code{,}' characters while

layer parameter such as activation function and number of neurons are separated by '\code{|}'

characters.

Let's see in detail how to build the layout string for the different models..

\paragraph{Dense Layer Layout}

For example the layout string

\begin{tmvacode}

Layout=TANH|128,TANH|128,TANH|128,LINEAR

\end{tmvacode}

defines a network with three hidden layers with 128 neurons and

$tanh$ activation functions. The neurons of the output layer are

inferred automatically from the number of classes (in the case of

classification) or the number of target variables (in the case of

regression).

The available activation functions are given in

Table~\ref{opt:mva:dnn:activations}.

\begin{option}[h]

\input optiontables/MVA__DNN_2.tex

\caption[.]{\optionCaptionSize

DL Activation Functions

}

\label{opt:mva:dnn:activations}

\end{option}

\subsubsection{Convolutional Networks \index{Convolutional Networks}}

\label {sec:cnn}

Convolutional Neural Networks are a popular variation of the feed-forward networks described in the previous section. By making the explicit assumption that the input is an image, they manage to

outperform conventional networks by significantly reducing the number of learnable parameters through a feature known as parameter sharing~\cite{alexnet}.

Instead of connecting every neuron of the current layer with every neuron of the previous layer as in conventional dense layers, we only learn the parameters for a set of a small kernels (typically

3x3 or 5x5) which slide over the input. The size of each kernel, as well as the number of pixels for each vertical or horizontal shift, are user defined meta-parameters. An important detail is that

while the model assumes spatial invariance with respect to the height and width, no such assumption is made with respect to the mage depth. This asymmetry implies that at each receptive field, the kernel is connected with every slice of the input volume.

In order to reduce overfitting, we sometimes include a pooling layer between subsequent convolutional layers. This layer simply downsamples the input, typically by outputting the maximum value within

its receptive field. This does not only reduce the number of neurons, but also helps reduce overfitting. For more details the reader might find this paper~\cite{maxpool} informative. The meta-parameters of this

layer type are a subset of those of a convolutional layer, and are described in the section below.

\paragraph{Convolutional Network Layout}

A typical convolution network will first include several convolutional layers with different meta-parameters, each of them possibly followed by max-pooling layers to reduce overfitting. The last layers are fully connected (dense), in order to facilitate a soft-max activation similarly to traditional ANNs. An example layout configuration is depicted in the snippet below:

\begin{tmvacode}

Layout=CONV|12|3|3|1|1|1|1|RELU, CONV|12|3|3|1|1|1|1|RELU, MAXPOOL|2|2|2|2,

RESHAPE|FLAT,DENSE|64|RELU,DENSE|1|LINEAR

\end{tmvacode}

Here we define a layout consisting of 2 convolutional layers, the last of which is followed by a pooling layer. After those layers, we include 2 fully connected layers where the last one consists of a

single neuron (binary classification). We also employ a flattening reshape layer necessary to connect the multi-dimensional output, produced by convolutional layers, to a dense layer.

Figure~\ref{fig''cnn_layer_layout} visually explains the layout parameterization of the first convolutional layer.

\begin{figure}

\begin{center}

\includegraphics[width=0.9\textwidth]{plots/cnn-layer-layout.pdf}

\label{fig:cnn_layer_layout}

\caption{}

\end{center}

\end{figure}

\paragraph{Convolutional Layer Layout}

The parameters of the Convolutional layers are defined in the Layout string and they are separated by the '\code{|}' delimiter character.

Note that all of them must be provided.

The parameters are the following, in order of occurence :

\begin{itemize}

\item Layer Type (string) : {\tt CONV} for defining a 2D convolutional layer.

\item Number of kernels (integer), {\it e.g.} 12 in the above example.

\item Filter height (integer).

\item Filter width (integer).

\item Vertical stride (integer). Note that the stride must be a divisor of the input image size.

\item Horizontal stride (integer).

\item Vertical padding (integer) The padding value has to be a valid one, i.e. it provides an output image size which is an integer and not fractional. See the formula below defining the output size

\item Horizontal padding (integer)

\item Activation function (string) : See Table~\ref{opt:mva:dnn:activations} for the list of possible activation functions.

\end{itemize}

The Padding and stride parameter values must be consistent and have values that used with the given input image sizes and filter sizes result in integer output sizes and not fractional.

The formula that defines the output size is:

\begin{equation}

\end{equation}

where $o$ is the horizontal (or vertical) output size, $i$, $f$,$p$, $s$ are the corresponding input, filter, padding and stride sizes.

In the example above with $f=3$, $s=1$,$p=1$ in both dimensions, the output image size will be equal to the input image size.

\paragraph{MaxPool Layer Layout}

The MaxPool layer is typically used in combination with convolutional layers in a CNN to down-sample the images.

The MaxPool layer computes the maximum of the input in the provided rectangular pool size.

The parameters describing the MaxPool layer are:

\begin{itemize}

\item Layer Type (string) : {\tt MAXPOOL} for 2D max pooling layer.

\item Pool height dimension (integer) : {\it e.g.} 2 in the example above.

\item Pool width dimension (integer).

\item Vertical stride (integer).

\item Horizontal stride (integer).

\end{itemize}

Note that the output images after a maxpooling will be reduced(down-sampled) according to the following formula:

\begin{equation}

\end{equation}

\paragraph{Reshape Layer}

The reshape layer is used to flatten the image that is the output from a convolutional layer or a maxpool layer in a format that can be input by a dense layer. After a convolution layer the output

data will have a dimension equal to batch size $\times$ number of kernels $\times$ output height $\times$ output width ($B \times C \times H \times W$). The reshape will reformat the data to a

2-dimensional tensor (i.e. a matrix) with batch size as number of rows and

number of kernels $\times$ height $\times$ width as number of columns ( $B \times CHW$) that can be processed by a dense layer.

The reshale layer can also perform the inverse operation (de-flattening)

The parameters of the reshape layers are different depending on the flattening or de-flattening case. In the flattening case the output depth, height and width can be omitted

\begin{itemize}

\item Layer Type (string) : {\tt RESHAPE}.

\item 1st output dimension (integer), e.g. output depth.

\item 2nd output dimension (integer).

\item 3rd output dimension (integer).

\item string type: {\tt FLAT} for a flattening layer. In that case no need to specify the output dimension.

\end{itemize}

\paragraph{Batch Normalization Layer}

The batch normalization layer normalizes the input by re-centering and re-scaling using the average and the standard deviation of the input batch. It is described in this paper~\cite{batch-norm}.

The parameters for this layer are:

\begin{itemize}

\item Layer type: {\tt BNORM}.

\item Momentum $\rho$, used for the running average (floating). Optional, default is $\rho = 0.99$.

\item $\epsilon$ parameter used for computing batchnorm (see formula in ~\cite{batch-norm}. Optional, default is $\epsilon=0.001$.

\end{itemize}

\subsubsection{Recurrent Neural Networks \index{Recurrent Networks}}

\label {sec:rnn}

Recurrent neural network is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic

behavior. Different types of recurrent networks exist.

This implementation provides the support for 3 types of recurrent layers:

\begin{itemize}

\item Simple Recurrent layer (Vanilla RNN) as described for example in par. 10.2 of ~\cite{deeplearning_book}.

\item LSTM (Long Short-Term Memory) layer as defined in ~\cite{lstm}

\item GRU (Gate Recurrent unit) layer as defined in ~\cite{gru}

\end{itemize}

Here is an example of defining a recurrent neural networks composed of one LSTM layer followed by a dense layer for classification:

\begin{tmvacode}

InputLayout=10|3:

Layout=LSTM|12|3|10|0|1|RESHAPE|FLAT,DENSE|64|RELU,DENSE|1|LINEAR

\end{tmvacode}

The layer layout parameters for recurrent layers are

\begin{itemize}

\item Layer Type (string) : {\tt RNN} (Vanilla RNN), {\tt LSTM}, {\tt GRU}.

\item Layer state size (integer).

\item Input size (integer)

\item Time steps size (integer), {\it i.e.} number of recurrent cells.

\item Remember state option (boolean), default is false (0). This corresponds to the stateful option in Keras.

\item Return the full output time sequence (boolean): default is false (0), i.e. return only output of last recurrent cell.

\item Reset gate after option (boolean) : default is false. This is an option that applies only for GRU which defines a slightly different implementation..

This option is forced to be true when using the GPU implementation, since it is the way cuDNN implements GRU. Note that with this option the GRU computation is slightly faster.

\end{itemize}

\subsubsection{Input Layout}

\label{sec:dnn:inputlayout}

When using a convolutional or a recurrent network it is needed to provide an input layout string defining the format of the input, i..e. .its shape. In the case of convolutional networks the size

of the input image (e.g. number of channels, image height, image width) must be defined and in the case of

recurrent networks we the time and the feature dimensions must be provided.

In the case of simple dense layer there is no need for an input layout because the total number of input event features can be deduced internally by TMVA..

The input layout shape is defined as following; here we give the example for input image with 3 channels, height = 32, width = 32:

\begin{tmvacode}

InputLayout=3|32|32

\end{tmvacode}

The input layout parameters (input tensor dimensions) are separated as in the case of the layer layout by the '\code{|}' character.

For a RNN it is enough to provide an input layout of 2 dimensions: time and feature size.

For example for an RNN with an input dimension of 3 features in 10 time steps, we will have an input layout as following

\begin{tmvacode}

InputLayout=10|3

\end{tmvacode}

\subsubsection{Training Strategy}

\label{sec:dnn:training}

The training of deep neural networks is performed in one or several

consecutive training phases that are specified using the

\code{TrainingStategy} argument of the \code{<options>} string passed

to the \code{BookMethod} call. A single phase is specified by the

parameters described in Table~\ref{opt:mva:dnn:training}. Different

training phases are separated by \code{|} characters.

As an example, a valid training strategy string would be the following:

\begin{tmvacode}

TrainingStrategy = LearningRate=1e-1,MaxEpoch=20

| LearningRate=1e-2, MaxEpoch=30

| LearningRate=1e-3, MaxEpoch=40

\end{tmvacode}

This training strategy trains the network in three training phases

each with a lower learning rate. This is particular useful if using the

$SGD$ optimizers which requires a manual adaptation of the learning rate.

When using instead an optimizer, such as $ADAM$, the learning rate adapts automatically and there

is no such need to define several training strategies.

Dropout is a regularization technique that with a certain probability

sets neuron activations to zero. This probability can be set for all

layers at once by giving a single floating point value or a value for

each hidden layer of the network separated by '\code{+}'

signs. Probabilities should be given by a value in the interval

$[0,1]$.

\begin{option}[h]

\input optiontables/MVA__DNN_3.tex

\caption[.]{\optionCaptionSize

Configuration options reference for MVA method: {\em DL}.

}

\label{opt:mva:dnn:training}

\end{option}