neoml.Dnn¶
Neural network¶
A neural network is a directed graph with the vertices corresponding to layers and the arcs corresponding to the connections along which the data is passed from one layer’s output to another’s input.
Each layer should be added to the network after you assign a unique name to it. A layer may not be connected to several networks at once.
Source layers are used to pass the data into the network. A source layer has no inputs and passes the data blob specified by the user to its only output.
Sink layers with no outputs are used to retrieve the result of the network operation. They provide a function that returns the blob with data.
After all the layers are added and connected the network may be set up for training.

class
neoml.Dnn.
Dnn
(math_engine, random=None)¶ Neural network implementation. A neural network is a directed graph consisting of layers that perform calculations on data blobs. It starts with source layers and ends with sink layers.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – The math engine that will perform calculations.
random (object, default=None) – The random numbers generator to be used for training and initialization.

add_layer
(layer)¶ Adds a layer to the network.
 Parameters
layer (neoml.Dnn.Layer) – the layer to be added.

delete_layer
(layer)¶ Deletes a layer from the network.
 Parameters
layer (neoml.Dnn.Layer or str) – the layer to be deleted, or its name

property
initializer
¶ The initializer that will fill in the weight values before training starts. Xavier initialization is the default.

property
input_layers
¶ All source layers of the network.

property
layers
¶ Gets all layers of the network.

learn
(inputs)¶ Runs the network, performs a backward pass and updates the trainable weights.
 Parameters
inputs (dict) – The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.

load
(path)¶ Loads the network from file.
 Parameters
path (str) – The full path to the location from where the network should be loaded.

load_checkpoint
(path)¶ Loads the checkpoint from file. A new solver will be created, because the old pointers will point to an object no longer used by this network.
 Parameters
path (str) – The full path to the location from where the network should be loaded.

property
math_engine
¶ The math engine neoml.MathEngine.MathEngine used by the network.

property
output_layers
¶ All sink layers of the network.

run
(inputs)¶ Runs the network.
 Parameters
inputs – The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.
 Type
inputs: dict

run_and_backward
(inputs)¶ Runs the network and performs a backward pass with the input data.
 Param
inputs: The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.

property
solver
¶ The optimizer for the layer’s trainable parameters.

store
(path)¶ Serializes the network.
 Parameters
path (str) – The full path to the location where the network should be stored.

store_checkpoint
(path)¶ Serializes the network with the data required to resume training.
 Parameters
path (str) – The full path to the location where the network should be stored.
Data blobs¶
All data used in the network operation (inputs, outputs, trainable parameters) is stored in blobs. A blob is a 7dimensional array, and each of its dimensions has a specific meaning:
BatchLength is a “time” axis, used to denote data sequences; it is mainly used in recurrent networks
BatchWidth corresponds to the batch, used to pass several independent objects together
ListSize is the dimensions for the objects that are connected (for example, pixels out of one image) but do not form a sequence
Height is the height of a matrix or an image
Width is the width of a matrix or an image
Depth is the width of a 3dimensional image
Channels corresponds to channels for multichannel image formats and is also used to work with onedimensional vectors.
The blobs may contain one of the two types of data: float
and int
. Both data types are 32bit.
If the data type is not specified directly anywhere in this documentation, that means float
is used.
Class description¶

class
neoml.Blob.
Blob
(internal)¶ The class that stores and transmits data in neural networks.

asarray
(copy=False, keep_dims=False)¶ Returns the contents of the blob as a multidimensional array.
 Parameters
copy (bool, default=False) – if True, the data will be copied. If False, the array may share the memory buffer with the blob if possible and only provide more convenient access to the same data. Not copying may be impossible if the blob is in GPU memory.
keep_dims – if True then the result shape will contain all the 7
dimensions of the blob. Otherwise it will contain only dimensions longer than 1. If False and blob consists only of one element, then the result will be of shape (1,). :type keep_dims: bool, default=False

property
batch_len
¶ The BatchLength dimension.

property
batch_width
¶ The BatchWidth dimension.

property
channels
¶ The Channels dimension.

copy
(math_engine)¶ Creates a blob copy independent of this blob.

property
depth
¶ The Depth dimension.

property
height
¶ The Height dimension.

property
list_size
¶ The ListSize dimension.

property
math_engine
¶ The math engine that allocates the blob memory and will work with its data.

property
object_count
¶ The number of objects in the blob, equal to BatchLength * BatchWidth * ListSize.

property
object_size
¶ The size of one object, equal to Height * Width * Depth * Channels.

property
shape
¶ The blob shape: the array of 7 int elements with dimension lengths.

property
size
¶ The blob total size: the product of all dimensions.

property
width
¶ The Width dimension.

Working with blobs¶

Blob.
store
(file_path)¶ Stores the blob in a file at the specified path.
 Parameters
file_path (str) – the full path to the file where the blob should be stored.

Blob.
load
(file_path)¶ Loads the blob from the specified location.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
file_path (str) – the full path to the file from which the blob should be loaded.

Blob.
asblob
(data, shape=None, copy=False)¶ Organizes the data from a memory buffer into a blob.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
data (object) – a pointer to the data.
shape (array of int, default=None) – the target blob dimensions. If none are specified, a oneelement blob will be assumed.
copy (bool, default=False) – specifies if the data should be copied to another memory block or kept in the same place if possible, making the data parameter point to the body of the newlycreated blob. Not copying may be impossible if the blob is in GPU memory.
Creating blobs of typical size¶
The auxiliary methods that create blobs for frequently used types of data.

Blob.
vector
(size, dtype='float32')¶ Creates a onedimensional blob, that is, a vector.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
size (int, > 0) – the vector length.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.
matrix
(matrix_height, matrix_width, dtype='float32')¶ Creates a twodimensional blob, that is, a matrix.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
matrix_height (int, > 0) – the matrix height.
matrix_width (int, > 0) – the matrix width.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.
tensor
(shape, dtype='float32')¶ Creates a blob of the specified shape.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
shape (array of int) – the target blob dimensions.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.
list_blob
(batch_len, batch_width, list_size, channels, dtype='float32')¶ Creates a blob with onedimensional Height * Width * Depth elements.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
list_size (int, > 0) – the ListSize dimension of the new blob.
channels (int, > 0) – the Channels dimension of the new blob.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.
image2d
(batch_len, batch_width, height, width, channels, dtype='float32')¶ Creates a blob with twodimensional multichannel images.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
height (int, > 0) – the image height.
width (int, > 0) – the image width.
channels (int, > 0) – the number of channels in the image format.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.
image3d
(batch_len, batch_width, height, width, depth, channels, dtype='float32')¶ Creates a blob with threedimensional multichannel images.
 Parameters
math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
height (int, > 0) – the image height.
width (int, > 0) – the image width.
depth (int, > 0) – the image depth.
channels (int, > 0) – the number of channels in the image format.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.
Layers¶
A layer is an element of the network that performs some operation: anything from the input data reshape or a simple math function calculation, up to convolution or LSTM (Long shortterm memory).
If the operation needs input data, it will be taken from the layer input. Each layer input contains one data blob, and if several blobs are needed, the layer will have several inputs. Each layer input should be connected to another layer’s output.
If the operation returns results that should be used by other layers, they will be passed to the layer outputs. Each layer output contains one data blob, so depending on the operation it performs the layer may have several outputs. Several other layer inputs may be connected to the same output, but you may not leave an output unconnected to any inputs.
In addition, the layer may have settings specified by the user before starting calculations, and trainable parameters that are optimized during network training.
The layers also have names that can be used to find a layer in the network. The name should be set at layer creation or before adding it to the network.
Base layer class¶
All NeoML layer classes are derived from this class.

class
neoml.Dnn.
Layer
(internal)¶ The base class for a network layer.

connect
(layer, output_index=0, input_index=0)¶ Connects this layer to another.
 Parameters
layer (neoml.Dnn.Layer) – the layer to which this one will be connected.
output_index (int, >=0, default=0) – the number of the layer’s output to be connected.
input_index (int, >=0, default=0) – the number of this layer’s input to be connected.

property
input_links
¶ Tuple in which i’th element contains complete information about the link between i’th input layer and self (layer name, output idx)

property
input_names
¶ Tuple in which i’th element contains the name of the layer, connected to the i’th inputs of self

property
learning_enabled
¶ Gets whether the weights are frozen or not.

property
name
¶ Gets the layer name.

Passing the data to and from the network¶
Source¶

class
neoml.Dnn.
Source
(dnn, name=None)¶ The source layer that serves to pass a user data blob into the network.
 Parameters
dnn (object) – The neural network.
name (str, default=None) – The layer name.
Layer inputs:
The layer has no inputs.
Layer outputs:
The layer has one output that contains the data blob passed into the last call of set_blob.

get_blob
()¶ Gets the blob with source data.

set_blob
(blob)¶ Sets the blob with source data.
Sink¶

class
neoml.Dnn.
Sink
(input_layer, name=None)¶ The sink layer that serves to pass a data blob out of the network. Use the
get_blob
method to retrieve the blob. Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
The layer has one input that accepts a data blob of any size.
Layer outputs:
The layer has no outputs.

get_blob
()¶ Gets the data blob.
Recurrent layers¶
Lstm¶
The long shortterm memory layer.

class
neoml.Dnn.
Lstm
(input_layer, hidden_size=1, dropout_rate=0.0, activation='sigmoid', reverse_seq=False, name=None)¶ A long shortterm memory (LSTM) layer that can be applied to a set of vector sequences. The output is a sequence containing the same number of vectors, each of the layer hidden size length.
 Parameters
input_layer (object, tuple(object, int) or list of them) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size and the LSTM state vector size.
dropout_rate (float, default=0.0) – The dropout probability for the input and the recurrent data.
recurrent_activation (str, {"linear", "elu", "relu", "leaky_relu", "abs", "sigmoid", "tanh", "hard_tanh", "hard_sigmoid", "power", "hswish", "gelu"}, default="sigmoid") – The activation function that is used in forget, reset, and input gates.
reverse_seq (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.
name (str, default=None) – The layer name.
Layer inputs:
the set of vector sequences, of dimensions:
BatchLength  the length of a sequence
BatchWidth * ListSize  the number of vector sequences in the input set
Height * Width * Depth * Channels  the size of each vector in the sequence
(optional): the initial state of the LSTM before the first step. If this input is not specified, the initial state is all zeros. The dimensions:
BatchLength should be 1
the other dimensions should be the same as for the first input
(optional): the initial value of “previous output” to be used on the first step. If this input is not specified, all zeros will be used. The dimensions:
BatchLength should be 1
the other dimensions should be the same as for the first input
Layer outputs:
the layer output on every step
(optional): the cell state on every step
Both outputs have the following dimensions:
BatchLength, BatchWidth, ListSize are equal to the first input dimensions
Height, Width, Depth are equal to 1
Channels is equal to layer hidden size

property
activation
¶ Gets the activation function.

property
dropout
¶ Gets the dropout rate.
Gets the hidden layer size.

property
input_free_term
¶ Gets the input free term.

property
input_weights
¶ Gets the input hidden layer weights. The blob size is (4*HiddenSize)x1x1xInputSize.

property
recurrent_free_term
¶ Gets the recurrent free term.

property
recurrent_weights
¶ Gets the recurrent hidden layer weights. The blob size is (4*HiddenSize)x1x1xHiddenSize.

property
reverse_sequence
¶ Checks if the input sequence will be reverted.
Qrnn¶

class
neoml.Dnn.
Qrnn
(input_layers, pooling_type='f', hidden_size=1, window_size=1, stride=1, paddings=(0, 0), activation='tanh', dropout=0.0, mode='direct', name=None)¶ The quasirecurrent layer that can be applied to a set of vector sequences. Unlike LSTM or GRU, the layer performs most of the calculations before the recurrent part, which helps improve performance on GPU. We use time convolution outside of the recurrent part instead of fullyconnected layers in the recurrent part. See https://arxiv.org/abs/1611.01576
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
pooling_type (str, {"f", "fo", "ifo"}) – The pooling type used after recurrent
hidden_size (int, > 0) – The hidden layer size.
window_size (int, > 0) – The size of the window used in time convolution.
stride (int, > 0, default=1) – The window stride for time convolution.
paddings (tuple(int, int), >= 0, default=(0, 0)) – The additional zeros tacked to the start and end of sequence before convolution.
activation (str, {"linear", "elu", "relu", "leaky_relu", "abs", "sigmoid", "tanh", "hard_tanh", "hard_sigmoid", "power", "hswish", "gelu"}, default="tanh") – The activation function used in the update gate.
dropout (float, [0..1], default=0.0) – The dropout probability in the forget gate.
mode (str, {"direct", "reverse", "bidirectional_concat", "bidirectional_sum"}, default="direct") – The way of processing the input sequences.  bidirectional_concat means the direct and the reverse sequence are concatenated and then processed as one;  bidirectional_sum means the direct and the reverse sequence are added up and then processed as one.
name (str, default=None) – The layer name.
Layer inputs:
the set of vector sequences, of dimensions:
BatchLength is the length of one sequence
BatchWidth is the number of vector sequences in the set
ListSize is 1
Height * Width * Depth * Channels is the vector length
(optional): the initial state of the recurrent part. If not set, the recurrent part is all zeros before the first step. The dimensions:
BatchLength, ListSize, Height, Width, Depth are 1
BatchWidth is the same as for the first input
Channels is hidden_size
Layer outputs:
the result sequence. The dimensions:
BatchLength can be calculated from the input as (BatchLength + paddings[0] + paddings[1]  (window_size  1))/(stride + 1)
BatchWidth is the same as for the inputs
ListSize, Height, Width, Depth are 1
Channels is hidden_size for all recurrent modes except bidirectional_concat, when it is 2 * hidden_size

property
activation
¶ Gets the activation function used in the update gate.

property
dropout
¶ Gets the dropout probability for the forget gate.

property
filter
¶ Gets the trained weights for each gate. The blob dimensions:
BatchLength is 1
BatchWidth is 3 * hidden_size (contains the weights for each of the three gates in the order: update, forget, output)
Height is window_size
Width, Depth are 1
Channels is equal to the input’s Height * Width * Depth * Channels

property
free_term
¶ Gets the free term for all three gates in the same order. The blob size is 3 * hidden_size.
Gets the hidden layer size.

property
padding_back
¶ Gets the size of zero padding at the sequence end.

property
padding_front
¶ Gets the size of zero padding at the sequence start.

property
recurrent_mode
¶ Gets the sequence processing mode.

property
stride
¶ Gets the stride for time convolution.

property
window_size
¶ Gets the window size for time convolution.
Gru¶
The gated recurrent unit.

class
neoml.Dnn.
Gru
(input_layer, hidden_size, name=None)¶ The gated recurrent unit (GRU) layer that works with a set of vector sequences.
 Parameters
input_layer – The input layer and the number of its output. If no number is specified, the first output will be connected.
input_layer – object, tuple(object, int) or list of them
hidden_size (int) – The size of the hidden layer.
name (str, default=None) – The layer name.
Layer inputs:
the set of vector sequences. The dimensions:
BatchLength is sequence length
BatchWidth * ListSize is the number of sequences
Height * Width * Depth * Channels is vector size
(optional): the initial previous step result. If you do not connect this input all zeros will be used on the first step. The dimensions are the same as for the first input.
Layer outputs:
a vector sequence of the same length. The dimensions:
BatchLength, BatchWidth, ListSize equal to the input’s dimensions
Height, Width, Depth are 1
Channels is equal to hidden layer size

property
gate_free_term
¶ Gets the gate free terms as a blob of total 2 * hidden_size size.

property
gate_weights
¶ Gets the gate weights as a 2d matrix of the size:
BatchLength * BatchWidth * ListSize equal to 2 * hidden_size
Height * Width * Depth * Channels equal to this dimension of the input plus hidden_size
Gets the hidden layer size.

property
main_free_term
¶ Gets the output free terms as a blob of total hidden_size size.

property
main_weights
¶ Gets the output weights as a 2d matrix of the size:
BatchLength * BatchWidth * ListSize equal to hidden_size
Height * Width * Depth * Channels equal to this dimension of the input plus hidden_size
Irnn¶

class
neoml.Dnn.
Irnn
(input_layer, hidden_size=1, identity_scale=1.0, input_weight_std=0.001, reverse_seq=False, name=None)¶ IRNN implementation from this article: https://arxiv.org/pdf/1504.00941.pdf
It’s a simple recurrent unit with the following formula: \(Y_t = ReLU( FC_{input}( X_t ) + FC_{recur}( Y_t1 ) )\) where \(FC\) are fullyconnected layers.
The crucial point of this layer is weights initialization. The weight matrix of \(FC_{input}\) is initialized from N(0, input_weight_std), where input_weight_std is a layer parameter. The weight matrix of \(FC_{recur}\) is an identity matrix, multiplied by identity_scale parameter.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size.
identity_scale (float, default=1.) – The scale of identity matrix, used for the initialization of recurrent weights.
input_weight_std (float, default=1e3) – The standard deviation for input weights.
reverse_seq (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.
name (str, default=None) – The layer name.
Layer inputs:
the set of vector sequences, of dimensions:
BatchLength  the length of a sequence
BatchWidth * ListSize  the number of vector sequences in the input set
Height * Width * Depth * Channels  the size of each vector in the sequence
Layer outputs:
the result of the current step. The dimensions:
BatchLength, BatchWidth, ListSize are the same as for the input
Height, Width, Depth are 1
Channels is hidden_size
Gets the hidden layer size.

property
identity_scale
¶ Gets the multiplier for the identity matrix for initialization.

property
input_free_term
¶ Gets the FC_input free term, of hidden_size length.

property
input_weight_std
¶ Gets the standard deviation for input weights.

property
input_weights
¶ Gets the FC_input weights. The dimensions:
BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is the same as for the input

property
recurrent_free_term
¶ Gets the FC_recur free term, of hidden_size length.

property
recurrent_weights
¶ Gets the FC_recur weights. The dimensions:
BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is hidden_size

property
reverse_sequence
¶ Checks if the input sequence should be taken in reverse order.
IndRnn¶

class
neoml.Dnn.
IndRnn
(input_layer, hidden_size=1, dropout_rate=0.0, reverse_sequence=False, activation='relu', name=None)¶ Independently Recurrent Neural Network (IndRNN): https://arxiv.org/pdf/1803.04831.pdf
It’s a simple recurrent unit with the following formula: \(Y_t = activation(W * X_t + B + U * dropout(Y_{t1}))\) where \(W\) and \(B\) are weights and free terms of the fullyconnected layer (\(W * X_t\) is a matrix multiplication) and \(U\) is a vector (\(U * Y_{t1}\) is an eltwise multiplication of 2 vectors of the same length)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size.
dropout_rate (float, default=0.) – The rate of the dropout, applied to both input and recurrent data
activation (str, {'sigmoid', 'relu'}, default='relu') – The activation funciton applied to the output
reverse_sequence (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.
Layer inputs:
the set of vector sequences, of dimensions:
BatchLength  the sequence length
BatchWidth * ListSize  the number of vector sequences in the input set
Height * Width * Depth * Channels  the size of each vector in the sequence
Layer outputs:
the result of the layer. The dimensions:
BatchLength, BatchWidth, ListSize are the same as for the input
Height, Width, Depth are 1
Channels is hidden_size

property
activation
¶ Gets the activation function.

property
bias
¶ Gets the bias vector, of hidden_size length.

property
dropout_rate
¶ Gets the dropout rate.
Gets the hidden layer size.

property
input_weights
¶ Gets the input weights matrix. The dimensions:
BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is the same as for the input

property
recurrent_weights
¶ Gets the recurrent weights vector, of hidden_size length.

property
reverse_sequence
¶ Checks if the input sequence should be taken in reverse order.
FullyConnected¶

class
neoml.Dnn.
FullyConnected
(input_layers, element_count, is_zero_free_term=False, name=None)¶ The fully connected layer. It multiplies each of the input vectors by the weight matrix and adds the free term vector to the result.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
element_count (int, > 0) – The length of each vector in the output.
is_zero_free_term (bool, default=False) – If True, the free term vector is set to all zeros and not trained. If False, the free term is trained together with the weights.
name (str, default=None) – The layer name.
Layer inputs:
The layer can have any number of inputs. The dimensions:
BatchLength * BatchWidth * ListSize is the number of vectors
Height * Width * Depth * Channels is the vector size; should be the same for all inputs
Layer outputs:
The layer returns one output for each input. The dimensions:
BatchLength, BatchWidth, ListSize the same as for the input
Height, Width, Depth are 1
Channels is element_count

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
element_count
¶ Gets the length of each vector in the output.

property
free_term
¶ Gets the free term vector, of element_count length.

property
weights
¶ Gets the trained weights as a blob of the dimensions:
BatchLength * BatchWidth * ListSize equal to element_count
Height, Width, Depth, Channels the same as for the first input

property
zero_free_term
¶ Sets the length of each vector in the output.
Activation layers¶
These layers calculate the value of different activation functions on their inputs.
Linear¶

class
neoml.Dnn.
Linear
(input_layer, multiplier, free_term, name=None)¶ The layer that calculates a linear activation function for each element of a single input: \(f(x) = multiplier * x + free\_term\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
multiplier (float) – The linear function multiplier.
free_term (float) – The linear function free term.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
free_term
¶ Gets the free term.

property
multiplier
¶ Gets the multiplier.
ELU¶

class
neoml.Dnn.
ELU
(input_layer, alpha, name=None)¶ The layer that calculates the ELU activation function for each element of the single input:
\(f(x) = alpha * (e^x  1)\) if \(x < 0\)
\(f(x) = x\) if \(x \ge 0\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
alpha (float) – The multiplier before the exponential function used for negative values of x.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
alpha
¶ Gets the exponent multiplier.
ReLU¶

class
neoml.Dnn.
ReLU
(input_layer, threshold=0.0, name=None)¶ The layer that calculates the ReLU activation function for each element of the single input:
\(f(x) = 0\) if \(x \le 0\)
\(f(x) = x\) if \(x > 0\)
You also can set the cutoff upper threshold:
\(f(x) = 0\) if \(x \le 0\)
\(f(x) = x\) if \(0 < x < threshold\)
\(f(x) = threshold\) if \(threshold \le x\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
threshold (float, default=0) – The upper cutoff threshold. 0 resets the threshold.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
threshold
¶ Gets the upper cutoff threshold.
LeakyReLU¶

class
neoml.Dnn.
LeakyReLU
(input_layer, alpha, name=None)¶ The layer that calculates the “leaky” ReLU activation function for each element of the single input:
\(f(x) = alpha * x\) if \(x \le 0\)
\(f(x) = x\) if \(x > 0\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
alpha (float, default=0) – The multiplier used for negative values of x.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
alpha
¶ Gets the multiplier used for negative values of x.
HSwish¶

class
neoml.Dnn.
HSwish
(input_layer, name=None)¶ The layer that calculates the HSwish activation function for each element of the single input:
\(f(x) = 0\) if \(x \le 3\)
\(f(x) = x * (x + 3) / 6\) if \(3 < x < 3\)
\(f(x) = x\) if \(x \ge 3\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
GELU¶

class
neoml.Dnn.
GELU
(input_layer, name=None)¶ The layer that calculates the GELU activation function for each element of the single input: \(f(x) = x / (1 + e^{1.702 * x})\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
Abs¶

class
neoml.Dnn.
Abs
(input_layer, name=None)¶ The layer that calculates the absolute value of each element of the single input.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
Sigmoid¶

class
neoml.Dnn.
Sigmoid
(input_layer, name=None)¶ The layer that calculates the sigmoid activation function for each element of the signle input: \(f(x) = 1 / (1 + e^{x})\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
Tanh¶

class
neoml.Dnn.
Tanh
(input_layer, name=None)¶ The layer that calculates the tanh activation function for each element of the single input: \(f(x) = (e^{2 * x}  1) / (e^{2 * x} + 1)\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
HardTanh¶

class
neoml.Dnn.
HardTanh
(input_layer, name=None)¶ The layer that calculates the HardTanh activation function for each element of the single input:
\(f(x) = 1\) if \(x \le 1\)
\(f(x) = x\) if \(1 < x < 1\)
\(f(x) = 1\) if \(x \ge 1\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements
HardSigmoid¶

class
neoml.Dnn.
HardSigmoid
(input_layer, slope, bias, name=None)¶ The layer that calculates the “hard sigmoid” activation function for each element of the single input:
\(f(x) = 0\) if \(x \le bias / slope\)
\(f(x) = slope * x + bias\) if \(bias / slope < x < (1  bias) / slope\)
\(f(x) = 1\) if \(x \ge (1  bias) / slope\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
slope (float, > 0) – The slope of the linear component.
bias (float) – The shift of the linear component from 0.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
bias
¶ Gets the shift of the linear component from 0.

property
slope
¶ Gets the slope of the linear component.
Power¶

class
neoml.Dnn.
Power
(input_layer, exponent, name=None)¶ The layer that raises each element of the input to the given power.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
exponent (float) – The power to which the input elements will be raised.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with activation function values on each of the input elements

property
exponent
¶ Gets the power to which the input elements will be raised.
Exp¶

class
neoml.Dnn.
Exp
(input_layer, name=None)¶ The layer that calculates the exponent function for each element of the single input.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with exponent function values on each of the input elements
Log¶

class
neoml.Dnn.
Log
(input_layer, name=None)¶ The layer that calculates the logarithm function for each element of the single input.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with logarithm function values on each of the input elements
Erf¶

class
neoml.Dnn.
Erf
(input_layer, name=None)¶ The layer that calculates the error function for each element of the single input.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
a data blob of the same size as the input, with error function values on each of the input elements
Convolutional layers¶
The layers that perform various types of convolution.
Conv¶

class
neoml.Dnn.
Conv
(input_layers, filter_count=1, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), dilation_size=(1, 1), is_zero_free_term=True, name=None)¶ The layer that performs convolution on a set of twodimensional multichannel images.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size (tuple(int, int), default=(3, 3)) – Filter size (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size (tuple(int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
dilation_size (tuple(int, int), default=(1, 1)) – The step values for dilated convolution (vertical, horizontal). The default value of (1, 1) means no dilation.
is_zero_free_term (bool, default=True) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height  (1 + DilationHeight * (FilterHeight  1))) / StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width  (1 + DilationWidth * (FilterWidth  1))) / StrideWidth + 1
Depth is equal to 1
Channels is equal to the number of filters

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
dilation_size
¶ Gets the step values for dilated convolution.

property
filter
¶ Gets the filters. The dimensions:
BatchLength * BatchWidth * ListSize is filter_count
Height, Width are taken from filter_size
Depth, Channels are equal to the inputs’ dimensions

property
filter_count
¶ Gets the number of filters.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is filter_count.

property
padding_size
¶ Gets the padding size.

property
stride_size
¶ Gets the convolution stride.
Conv3D¶

class
neoml.Dnn.
Conv3D
(input_layers, filter_count=1, filter_size=(3, 3, 3), stride_size=(1, 1, 1), padding_size=(0, 0, 0), is_zero_free_term=False, name=None)¶ The layer that performs convolution on a set of threedimensional multichannel images.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int, int), default=(3, 3, 3)) – Filter size (height, width, depth).
stride_size ((int, int, int), default=(1, 1, 1)) – Convolution stride (vertical, horizontal, depth).
padding_size ((int, int, int), default=(0, 0, 0)) – The size of the padding (vertical, horizontal, depth).
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height  FilterHeight) / StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width  FilterWidth) / StrideWidth + 1
Depth can be calculated from the input Depth as (2 * PaddingDepth + Depth  FilterDepth) / StrideDepth + 1
Channels is equal to the number filters

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
filter
¶ Gets the filters. The dimensions:
BatchLength * BatchWidth * ListSize is filter_count
Height, Width, Depth are taken from filter_size
Channels is equal to the inputs’ Channels

property
filter_count
¶ Gets the number of filters.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is filter_count.

property
padding_size
¶ Gets the padding size.

property
stride_size
¶ Gets the convolution stride.
TransposedConv3D¶

class
neoml.Dnn.
TransposedConv3D
(input_layers, filter_count=1, filter_size=(3, 3, 3), stride_size=(1, 1, 1), padding_size=(0, 0, 0), is_zero_free_term=False, name=None)¶ The layer that performs transposed convolution on a set of threedimensional multichannel images.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int, int), default=(3, 3, 3)) – Filter size (height, width, depth).
stride_size ((int, int, int), default=(1, 1, 1)) – Convolution stride (vertical, horizontal, depth).
padding_size ((int, int, int), default=(0, 0, 0)) – The size of the padding (vertical, horizontal, depth).
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as StrideHeight * (Height  1) + FilterHeight  2 * PaddingHeight
Width can be calculated from the input Width as StrideWidth * (Width  1) + FilterWidth  2 * PaddingWidth
Depth can be calculated from the input Depth as StrideDepth * (Depth  1) + FilterDepth  2 * PaddingDepth
Channels is equal to the number of filters

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
filter
¶ Gets the filters. The dimensions:
BatchLength, ListSize are 1
BatchWidth is equal to the inputs’ Channels
Height, Width, Depth are taken from filter_size
Channels is filter_count

property
filter_count
¶ Gets the number of filters.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is filter_count.

property
padding_size
¶ Gets the padding size.

property
stride_size
¶ Gets the convolution stride.
TransposedConv¶

class
neoml.Dnn.
TransposedConv
(input_layers, filter_count=1, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), dilation_size=(1, 1), is_zero_free_term=False, name=None)¶ The layer that performs transposed convolution on a set of twodimensional multichannel images.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int), default=(3, 3)) – Filter size (height, width).
stride_size ((int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size ((int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
dilation_size ((int, int), default=(1, 1)) – The step values for dilated convolution (vertical, horizontal). The default value of (1, 1) means no dilation.
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as StrideHeight * (Height  1) + (FilterHeight  1) * DilationHeight + 1  2 * PaddingHeight
Width can be calculated from the input Width as StrideWidth * (Width  1) + (FilterWidth  1) * DilationWidth + 1  2 * PaddingWidths
Depth is 1
Channels is equal to the number of filters

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
dilation_size
¶ Gets the step values for dilated convolution.

property
filter
¶ Gets the filters. The dimensions:
BatchLength, ListSize are 1
BatchWidth is equal to the inputs’ Channels * Depth
Height, Width are taken from filter_size
Channels is filter_count

property
filter_count
¶ Gets the number of filters.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is filter_count.

property
padding_size
¶ Gets the padding size.

property
stride_size
¶ Gets the convolution stride.
ChannelwiseConv¶

class
neoml.Dnn.
ChannelwiseConv
(input_layers, filter_count, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), is_zero_free_term=False, name=None)¶ The layer that performs channelwise convolution. Each channel of the input is convolved with the corresponding channel of the filter.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of channels in the filter. Should be the same as the number of channels in the input.
filter_size ((int, int), default=(3, 3)) – Filter size (height, width).
stride_size ((int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size ((int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
free_term (bool, default=True) – Specifies if the free term should be nonzero.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height  FilterHeight)/StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width  FilterWidth)/StrideWidth + 1
Depth is equal to 1
Channels is equal to the number of channels in the filter and the input

apply_batch_normalization
(layer)¶ Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.
 Parameters
layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property
filter
¶ Gets the filter. The dimensions:
BatchLength, BatchWidth, ListSize, Depth are 1
Height, Width are taken from filter_size
Channels is equal to the inputs’ Channels

property
filter_count
¶ Gets the number of channels in the filter.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is inputs’ Channels.

property
padding_size
¶ Gets the padding size.

property
stride_size
¶ Gets the convolution stride.
TimeConv¶

class
neoml.Dnn.
TimeConv
(input_layers, filter_count, filter_size, padding_front=0, padding_back=0, dilation=1, stride=1, name=None)¶ The layer that performs time convolution on a set of sequences.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, > 0) – The number of filters.
filter_size (int, > 0) – Filter size.
padding_front (int, >= 0) – The number of zeros to be added at the sequence start.
padding_back (int, >= 0) – The number of zeros to be added at the sequence end.
dilation (int, default=1) – The step value for dilated convolution. 1 means no dilation.
stride (int, > 0, default=1) – Convolution stride.
name (str, default=None) – The layer name.
Layer inputs:
Can have several inputs, of the dimensions:
BatchLength is the sequence length
BatchWidth * ListSize  the number of sequences in the set
Height * Width * Depth * Channels  the size of each element
Layer outputs:
The layer has as many outputs as the inputs, of the dimensions:
BatchLength can be calculated from the input BatchLength as (padding_front + padding_back + BatchLength  (1 + dilation * (filter_size  1)))/stride + 1
BatchWidth, ListSize are equal to the input dimensions
Height, Width, Depth are equal to 1
Channels is equal to the number of filters

property
dilation
¶ Gets the step value for dilated convolution.

property
filter
¶ Gets the filters. The dimensions:
BatchLength is 1
BatchWidth is filter_count
Height is filter_size
Width, Depth are 1
Channels is the inputs’ Height * Width * Depth * Channels

property
filter_count
¶ Gets the number of filters.

property
filter_size
¶ Gets the filter size.

property
free_term
¶ Gets the free term. The blob size is filter_count.

property
padding_back
¶ Gets the padding size.

property
padding_front
¶ Gets the padding size.

property
stride
¶ Gets the convolution stride.
Loss layers¶
These layers calculate different types of loss functions on the network response. They have no output and return the loss value through a last_loss method.
CrossEntropyLoss¶

class
neoml.Dnn.
CrossEntropyLoss
(input_layers, softmax=True, loss_weight=1.0, name=None)¶ The layer that calculates the loss value as crossentropy between the result and the standard: \(loss = \sum{y_i * \log{z_i}}\), where for each i class \(y_i\) represents the class label, \(z_i\) is the network response (with softmax applied or not)
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
softmax (bool, default=True) – Specifies if softmax function should be applied to the result.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. Two formats are acceptable:
The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.

property
apply_softmax
¶ Checks if softmax function should be applied to the result.
BinaryCrossEntropyLoss¶

class
neoml.Dnn.
BinaryCrossEntropyLoss
(input_layers, positive_weight=1.0, loss_weight=1.0, name=None)¶ The layer that calculates the crossentropy loss function for binary classification: \(loss =  y * \log(sigmoid(x))  (1  y) * \log(1  sigmoid(x))\), where x is the network response, y is the correct class label (can be 1 or 1)
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
positive_weight (float, default=1.0) – The multiplier for correctly classified objects. Tune this value to prioritize precision (positive_weight < 1) or recall (positive_weight > 1) during training.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or 1).
(optional): the objects’ weights.
The dimensions of all inputs are the same:
BatchLength * BatchWidth * ListSize  the number of objects
Height, Width, Depth, Channels should be equal to 1
Layer outputs:
The layer has no output.

property
positive_weight
¶ Gets the multiplier for the term that corresponds to the correct results.
EuclideanLoss¶

class
neoml.Dnn.
EuclideanLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates the loss function equal to Euclidean distance between the network response and the correct classes.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss.
the correct class objects.
(optional): the objects’ weights.
The dimensions of all inputs are the same:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the object size
Layer outputs:
The layer has no output.
HingeLoss¶

class
neoml.Dnn.
HingeLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates hinge loss function for binary classification: \(f(x) = \max(0, 1  x * y)\), where x is the network response, y is the correct class label (1 or 1).
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or 1).
(optional): the objects’ weights.
The dimensions of all inputs are the same:
BatchLength * BatchWidth * ListSize  the number of objects
Height, Width, Depth, Channels should be equal to 1
Layer outputs:
The layer has no output.
SquaredHingeLoss¶

class
neoml.Dnn.
SquaredHingeLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates squared hinge loss function for binary classification:
\(f(x) = 4 * x * y\) if \(x * y < 1\)
\(f(x) = (\max(0, 1  x * y))^2\) if \(x * y \ge 1\)
where: x is the network response, y is the correct class label (1 or 1).
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or 1).
(optional): the objects’ weights.
The dimensions of all inputs are the same:
BatchLength * BatchWidth * ListSize  the number of objects
Height, Width, Depth, Channels should be equal to 1
Layer outputs:
The layer has no output.
FocalLoss¶

class
neoml.Dnn.
FocalLoss
(input_layers, force, loss_weight=1.0, name=None)¶ The layer that calculates the focal loss function for multiple class classification. It is a version of crossentropy in which the easilydistinguished objects receive smaller penalties. This helps focus on learning the difference between similarlooking elements of different classes.
\(f(x) = (1  x_{right})^{force} * \log(x_{right})\) where \(x_{right}\) is the network response element that represents the probability for the object to belong to the correct class.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
force (float, > 0, default=2.0) – The focal force, that is, the degree to which learning will concentrate on similar objects.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. Two formats are acceptable:
The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.

property
force
¶ Gets the focal force multiplier.
BinaryFocalLoss¶

class
neoml.Dnn.
BinaryFocalLoss
(input_layers, force, loss_weight=1.0, name=None)¶ The layer that calculates the focal loss function for binary classification. It is a version of crossentropy in which the easilydistinguished objects receive smaller penalties. This helps focus on learning the difference between similarlooking elements of different classes.
\(f(x) = (sigmoid(y * x))^{force} * \log(1 + e^{y * x})\) where: x is the network response, y is the correct class label (1 or 1).
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
force (float, > 0, default=2.0) – The focal force, that is, the degree to which learning will concentrate on similar objects.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or 1).
(optional): the objects’ weights.
The dimensions of all inputs are the same:
BatchLength * BatchWidth * ListSize  the number of objects
Height, Width, Depth, Channels should be equal to 1
Layer outputs:
The layer has no output.

property
force
¶ Gets the focal force multiplier.
CenterLoss¶

class
neoml.Dnn.
CenterLoss
(input_layers, class_count, rate, loss_weight=1.0, name=None)¶ The layer that penalizes large differences between objects of the same class. See the paper at http://ydwen.github.io/papers/WenECCV16.pdf
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
class_count (int) – The number of classes in the model.
rate (float, [0..1]) – Class center convergence rate: the multiplier used for calculating the moving mean of the class centers for each subsequent iteration.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. Two formats are acceptable:
The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.

property
class_count
¶ Gets the number of classes in the model.

property
rate
¶ Gets the convergence rate multiplier.
MultiHingeLoss¶

class
neoml.Dnn.
MultiHingeLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates hinge loss function for multiple class classification: \(f(x) = \max(0, 1  (x_{right}  x_{max\_wrong}))\) where \(x_{right}\) is the network response for the correct class, \(x_{max\_wrong}\) is the largest response for all the incorrect classes.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. Two formats are acceptable:
The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.
MultiSquaredHingeLoss¶

class
neoml.Dnn.
MultiSquaredHingeLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates squared hinge loss function for multiple class classification:
\(f(x) = 4 * (x_{right}  x_{max\_wrong})\) if \(x_{right}  x_{max\_wrong} < 1\)
\(f(x) = (\max(0, 1  (x_{right}  x_{max\_wrong})))^2\) if \(x_{right}  x_{max\_wrong} \ge 1\)
where \(x_{right}\) is the network response for the correct class, \(x_{max\_wrong}\) is the largest response for all the incorrect classes.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. Two formats are acceptable:
The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.
CustomLoss¶
NeoML provides an interface for userimplemented custom loss functions. They must be constructed out of simple arithmetic and Autodifferentiation functions.

class
neoml.Dnn.
CustomLossCalculatorBase
¶ The base class that you should implement to calculate the custom loss function.

abstract
calc
(data, labels)¶ Calculates the custom loss function. This function may use only the operations supported for autodiff:
simple arithmetic: +  * /
the neoml.AutoDiff.* functions
neoml.Autodiff.const for creating additional blobs filled with given values
 Parameters
data (neoml.Blob.Blob) – the network response with the probability
distribution of objects over classes. The blob dimensions:
BatchLength  the number of objects
Channels  the object size
all other dimensions equal to 1
 Parameters
labels (neoml.Blob.Blob) – the correct labels, of the same dimensions
as the first blob, and containing 1 in the coordinate of the class to which the corresponding object belongs, 0 in all other places.
 Returns
a blob that contains the loss function values
for each object in the batch. This blob will have the same BatchLength as the input blobs, and all its other dimensions should be 1. :rtype: neoml.Blob.Blob

abstract

class
neoml.Dnn.
CustomLoss
(input_layers, loss_calculator=None, loss_weight=1.0, name=None)¶ The layer that calculates a custom loss function.
 Parameters
input_layers (list of object, tuple(object, int)) – the input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_calculator (neoml.Dnn.CustomLossCalculatorBase) – a userimplemented object that provides the method to calculate the custom loss.
loss_weight (float, default=1.0) – the multiplier for the loss function value during training.
name (str, default=None) – the layer name.
Layer inputs:
the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
BatchLength * BatchWidth * ListSize  the number of objects
Height * Width * Depth * Channels  the number of classes
the correct class labels. The blob of the same dimensions as the first input, filled with zeros, where only the coordinate of the class to which the corresponding object from the first input belongs is be 1.
(optional): the objects’ weights. The dimensions:
BatchLength, BatchWidth, ListSize should be the same as for the first input
the other dimensions should be 1
Layer outputs:
The layer has no output.
Pooling layers¶
The layers that perform pooling operations on the input data.
MaxPooling¶

class
neoml.Dnn.
MaxPooling
(input_layers, filter_size=(3, 3), stride_size=(1, 1), name=None)¶ The pooling layer that finds maximum in a window.
 Parameters
input_layers ((object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int), default=(3, 3)) – The size of the window: (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Window stride (vertical, horizontal).
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
the result of pooling The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input as (Height  FilterHeight)/StrideHeight + 1
Width can be calculated from the input as (Width  FilterWidth)/StrideWidth + 1
Depth and Channels are equal to the input dimensions
MeanPooling¶

class
neoml.Dnn.
MeanPooling
(input_layers, filter_size=(3, 3), stride_size=(1, 1), name=None)¶ The pooling layer that takes average over the window.
 Parameters
input_layers ((object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int), default=(3, 3)) – The size of the window: (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Window stride (vertical, horizontal).
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
the result of pooling The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input as (Height  FilterHeight)/StrideHeight + 1
Width can be calculated from the input as (Width  FilterWidth)/StrideWidth + 1
Depth and Channels are equal to the input dimensions
GlobalMaxPooling¶

class
neoml.Dnn.
GlobalMaxPooling
(input_layers, max_count, name=None)¶ The layer that finds maximum over the whole threedimensional image, allowing for multiple largest elements to be found. If you set the number of largest elements to 1, it will function exactly as MaxPooling3d with the filter size equal to the input image size.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
max_count (int, > 0) – The number of largest elements to be found. Note that these do not have to be equal to each other; the top max_count elements will be returned.
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
the maximum values found. The dimensions:
BatchLength, BatchWidth, ListSize, Channels are equal to the input dimensions
Height, Depth are 1
Width is max_count
(optional): the indices of the values found in the input blob. The dimensions are the same.

property
max_count
¶ Gets the number of largest elements to be found.
GlobalMeanPooling¶

class
neoml.Dnn.
GlobalMeanPooling
(input_layers, name=None)¶ The layer that finds the average over the whole threedimensional image. It functions exactly as MeanPooling3d with the filter size equal to the input image size.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
the average values over each image. The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height, Width, Depth are 1
Channels is equal to the input Channels
GlobalSumPooling¶

class
neoml.Dnn.
GlobalSumPooling
(input_layers, name=None)¶ The layer that finds the sum over the whole threedimensional image.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
the sums over each image. The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height, Width, Depth are 1
Channels is equal to the input Channels
MaxOverTimePooling¶

class
neoml.Dnn.
MaxOverTimePooling
(input_layers, filter_len, stride_len, name=None)¶ The layer that finds maximums on the set of sequences, with the window taken over BatchLength axis.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_len (int) – The window size. Set to <= 0 if the maximum over the whole sequence length should be found.
stride_len (int, > 0) – The window stride. Meaningful only for filter_len > 0.
name (str, default=None) – The layer name.
Layer inputs:
the set of sequences, of dimensions:
BatchLength is the sequence length
BatchWidth * ListSize is the number of sequences in the set
Height * Width * Depth * Channels is the length of each vector
Layer outputs:
the pooling result. The dimensions:
BatchLength is:
1 if filter_len is <= 0
(BatchLength  filter_len) / stride_len + 1 otherwise
the other dimensions are the same as for the input

property
filter_len
¶ Gets the window size.

property
stride_len
¶ Gets the window stride.
ProjectionPooling¶

class
neoml.Dnn.
ProjectionPooling
(input_layers, dimension, original_size, name=None)¶ The layer that takes the average over one of the dimensions. This dimension is either compressed to a point when original_size=False, or stays the same length with all elements equal to the average when original_size=True.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – The dimension along which the average is taken.
original_size (bool, default=False) – Specifies if the blob should stay the same size, with the average value broadcast along the pooling dimension.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size.
Layer outputs:
the result of pooling. The dimensions:
all stay the same if original_size is True
if original_size is False, the pooling dimension is 1 and other dimensions stay the same

property
dimension
¶ Gets the dimension along which the average is to be calculated.

property
original_size
¶ Checks if the blob will stay the same size, with the average value broadcast along the pooling dimension.
MaxPooling3D¶

class
neoml.Dnn.
MaxPooling3D
(input_layers, filter_size=(3, 3, 3), stride_size=(1, 1, 1), name=None)¶ The pooling layer that finds maximum in a window for threedimensional images.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int, int), default=(3, 3, 3)) – The size of the window: (height, width, depth).
stride_size (tuple(int, int, int), default=(1, 1, 1)) – Window stride (vertical, horizontal, depth).
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
the result of pooling The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input as (Height  FilterHeight)/StrideHeight + 1
Width can be calculated from the input as (Width  FilterWidth)/StrideWidth + 1
Depth can be calculated from the input as (Depth  FilterDepth)/StrideDepth + 1
Channels is equal to the input Channels
MeanPooling3D¶

class
neoml.Dnn.
MeanPooling3D
(input_layers, filter_size=(3, 3, 3), stride_size=(1, 1, 1), name=None)¶ The pooling layer that takes average over a window for threedimensional images.
 Parameters
input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int, int), default=(3, 3, 3)) – The size of the window: (height, width, depth).
stride_size (tuple(int, int, int), default=(1, 1, 1)) – Window stride (vertical, horizontal, depth).
name (str, default=None) – The layer name.
Layer inputs:
the set of images, of dimensions:
BatchLength * BatchWidth * ListSize  the number of images in the set
Height  the images’ height
Width  the images’ width
Depth  the images’ depth
Channels  the number of channels the image format uses
Layer outputs:
the result of pooling The dimensions:
BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input as (Height  FilterHeight)/StrideHeight + 1
Width can be calculated from the input as (Width  FilterWidth)/StrideWidth + 1
Depth can be calculated from the input as (Depth  FilterDepth)/StrideDepth + 1
Channels is equal to the input Channels
Softmax¶

class
neoml.Dnn.
Softmax
(input_layer, area='object_size', name=None)¶ The layer that calculates softmax function on each vector of a set: softmax(x[0], … , x[n1])[i] = exp(x[i]) / (exp(x[0]) + … + exp(x[n1]))
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
area (str, {"object_size", "batch_length", "list_size", "channel"}, default="object_size") –
Specifies which dimensions constitute the vector length:
object_size: there are BatchLength * BatchWidth * ListSize vectors, of Height * Width * Depth * Channels length each
batch_length: there are BatchWidth * ListSize * Height * Width * Depth * Channels vectors, of BatchLength length each
list_size: there are BatchLength * BatchWidth * Height * Width * Depth * Channels vectors, of ListSize length each
channel: there are BatchLength * BatchWidth * ListSize * Height * Width * Depth vectors, of Channels length each
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size. The area setting determines which dimensions would be considered to constitute vector length.
Layer outputs:
a blob of the same size with softmax applied to every vector.

property
area
¶ Checks which dimensions constitute the vector length.
Dropout¶

class
neoml.Dnn.
Dropout
(input_layer, rate=0.5, spatial=False, batchwise=False, name=None)¶ The layer that randomly sets some elements of the single input to 0. If the input BatchLength is greater than 1, all elements along the same BatchLength coordinate will use the same mask. When the network is not being trained (for example, during a test run), the dropout will not happen.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
rate (float, [0..1]) – The proportion of elements that will be set to 0.
spatial (bool, default=False) – Turns on and off the spatial dropout mode. When True, the whole contents of a channel will be filled with zeros, instead of elements one by one. Useful for convolutional networks.
batchwise (bool, default=False) – Turns on and off the batchwise dropout mode. When True, the same mask will be used along the same BatchWidth. Useful for large input size.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any dimensions.
Layer outputs:
a blob of the same dimensions, with some of the elements set to 0, during training only. When you run the network, this layer does nothing.

property
batchwise
¶ Checks if the batchwise mode is on.

property
rate
¶ Gets the dropout rate.

property
spatial
¶ Checks if the spatial mode is on.
CumSum¶

class
neoml.Dnn.
CumSum
(input_layer, dimension='channels', reverse=False, name=None)¶ The layer that calculates cumulative sum along the given dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {'batch_length', 'batch_width', 'list_size', 'height', 'width', 'depth', 'channels'}, default='channels') – The dimension along which the cumulative sum is to be calculated.
reverse (bool, default=False) – If True then cumulative sums will be calculated in reverse order
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size and data type
Layer outputs:
a blob of the same size and data type with the cumulative sums

property
dimension
¶ Gets the dimension along which the cumulative sum is to be calculated.

property
reverse
¶ Gets the flag of reversed sum order
Normalization layers¶
BatchNormalization¶

class
neoml.Dnn.
BatchNormalization
(input_layer, channel_based=True, zero_free_term=False, slow_convergence_rate=1.0, name=None)¶ The layer that performs normalization using the formula: \(bn(x)[i][j] = ((x[i][j]  mean[j]) / \sqrt{var[j]}) * gamma[j] + beta[j]\)
gamma and beta are the trainable parameters
mean and var depend on whether the layer is being trained:
If the layer is being trained, mean[j] and var[j] are the mean value and the variance of x data with j coordinate across all i.
If the layer is not being trained, mean[j] and var[j] are the exponential moving mean and the unbiased variance estimate calculated during training.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
channel_based (bool, default=True) –
Turns on and off channelbased statistics:
If True, mean, var, gamma, and beta in the formula will be vectors of the input Channels length. The i coordinate will iterate over all values from 0 to BatchLength * BatchWidth * ListSize * Height * Width * Depth  1.
If False, the mean, var, gamma, and beta vectors will have the Height * Width * Depth * Channels length. The i coordinate will iterate over all values from 0 to BatchLength * BatchWidth * ListSize  1.
zero_free_term (bool, default=False) – Specifies if the free term (beta) should be trained or filled with zeros.
slow_convergence_rate (float, default=1.0) – The coefficient for calculating the exponential moving mean and variance.
name (str, default=None) – The layer name.

property
channel_based
¶ Gets the channelbased mode.

property
final_params
¶ Gets the trained parameters as a blob of the dimensions:
BatchLength, ListSize, Channels equal to 1
BatchWidth is 2
Height, Width, Depth equal to 1 if in channelbased mode, otherwise the same as the dimensions of the input

property
slow_convergence_rate
¶ Gets the coefficient for calculating the exponential moving mean and variance.

property
use_final_params_for_init
¶ Checks if the final parameters should be used for initialization.

property
zero_free_term
¶ Indicates if the free term should be zero.
ObjectNormalization¶

class
neoml.Dnn.
ObjectNormalization
(input_layer, epsilon=1e05, name=None)¶ The layer that performs object normalization using the formula: \(objectNorm(x)[i][j] = ((x[i][j]  mean[i]) / \sqrt{var[i] + epsilon}) * scale[j] + bias[j]\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
epsilon (float, default=0.00001) – The small value added to the variance to avoid division by zero.
name (str, default=None) – The layer name.
Layer inputs:
a set of objects. The dimensions:
BatchLength * BatchWidth * ListSize is the number of objects
Height * Width * Depth * Channels is the object size
Layer outputs:
the normalized result, of the same size as the input.

property
bias
¶ Gets bias, one of the trainable parameters in the formula. The total blob size is equal to the input Height * Width * Depth * Channels.

property
epsilon
¶ Gets the small value used to avoid division by zero.

property
scale
¶ Gets scale, one of the trainable parameters in the formula. The total blob size is equal to the input Height * Width * Depth * Channels.
Lrn (Local Response Normalization)¶

class
neoml.Dnn.
Lrn
(input_layer, window_size=1, bias=1.0, alpha=0.0001, beta=0.75, name=None)¶  Lrn layer performs local response normlization with the following formula:
\(LRN(x)[obj][ch] = x[obj][ch] * / ((bias + alpha * sqrSum[obj][ch] / windowSize) ^ beta)\) where \(obj\) is index of the object , \(ch\) is index of the channel, \(window_size\), \(bias\), \(alpha\) and \(beta\) are layer settings and :math:`sqrSum[obj][ch] = sum_{i=max(0, ch  lfloor
rac{windowSize  1}{2} floor)}^{min(C  1, ch + lceil rac{windowSize  1}{2} ceil)}x[obj][i]^2`
 param input_layer
The input layer and the number of its output. If no number is specified, the first output will be connected.
 type input_layer
object, tuple(object, int)
 param window_size
The size of window used in normalization
 type
int, default=1
 param bias
value added to the scaled sum of squares
 type
float, default=1.
 param alpha
scale value of sum of squares
 type
float, default=1e4
 param beta
exponent of the formula
 type
float, default=0.75
Layer inputs:
the set of objects, of dimensions:
BatchLength * BatchWidth * ListSize * Height * Width * Depth  the number of objects
Channels  the size of the object
Layer outputs:
the result of the layer, of the dimensions of the input.

property
alpha
¶ Gets the alpha.

property
beta
¶ Gets the beta.

property
bias
¶ Gets the bias.

property
window_size
¶ Gets the window size.
Elementwise operation layers¶
EltwiseSum¶

class
neoml.Dnn.
EltwiseSum
(input_layers, name=None)¶ The layer that adds up its inputs element by element.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have at least two inputs. All inputs should be the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
EltwiseSub¶

class
neoml.Dnn.
EltwiseSub
(input_layers, name=None)¶ The layer that substracts the second input from the first element by element.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have at least two inputs. All inputs should be the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
EltwiseMul¶

class
neoml.Dnn.
EltwiseMul
(input_layers, name=None)¶ The layer that multiplies its inputs element by element.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have at least two inputs. All inputs should be the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
EltwiseDiv¶

class
neoml.Dnn.
EltwiseDiv
(input_layers, name=None)¶ The layer that divides its inputs element by element.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have two inputs of the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
EltwiseNegMul¶

class
neoml.Dnn.
EltwiseNegMul
(input_layers, name=None)¶ The layer that that calculates the elementwise product of 1  x, where x is the element of the first input, and the corresponding elements of all other inputs.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have at least two inputs. All inputs should be the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
EltwiseMax¶

class
neoml.Dnn.
EltwiseMax
(input_layers, name=None)¶ The layer that finds the maximum among the elements that are at the same position in all input blobs.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer should have at least two inputs. All inputs should be the same size.
Layer outputs:
The layer has one output of the same size as the inputs.
Quality control layers¶
Accuracy¶

class
neoml.Dnn.
Accuracy
(input_layers, reset=True, name=None)¶ The layer that calculates classification accuracy, that is, the proportion of objects classified correctly in the set.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.
Layer inputs:
a blob with the network response The dimensions:
BatchLength * BatchWidth * ListSize equal to the number of objects that were classified
Height, Width, Depth equal to 1
Channels equal to 1 for binary classification and to the number of classes if there are more than 2
a blob with the correct class labels The dimensions should be the same as for the first input
Layer outputs:
a blob with only one element, which contains the proportion of correctly classified objects among all objects

property
reset
¶ Checks if the statistics will be reset after each run.
PrecisionRecall¶

class
neoml.Dnn.
PrecisionRecall
(input_layers, reset=True, name=None)¶ The layer that calculates the number of objects classified correctly for either class in a binary classification scenario. Using these statistics, you can easily calculate the precision and recall for the trained network.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.
Layer inputs:
the network response. The dimensions:
BatchLength * BatchWidth * ListSize is the number of objects classified
Height, Width, Depth, Channels are 1
the correct class labels (1 or 1). The dimensions are the same as for the first input.
Layer outputs:
the 4element array along the Channels dimension that contains:
0  the number of correctly classified objects in class 1
1  the total number of objects in class 1
2  the number of correctly classified objects in class 1
3  the total number of objects in class 1

property
reset
¶ Checks if the statistics will be reset after each run.

property
result
¶ Returns the result 4element array, in the same order as the output blob.
ConfusionMatrix¶

class
neoml.Dnn.
ConfusionMatrix
(input_layers, reset=True, name=None)¶ The layer that calculates the confusion matrix for classification results. The columns correspond to the network response, the rows  to the correct labels. Each element of the matrix contains the number of objects that belong to the “row” class and were classified as the “column” class.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.
Layer inputs:
a blob with the network response The dimensions:
BatchLength * BatchWidth * ListSize equal to the number of objects that were classified
Height, Width, Depth equal to 1
Channels equal to the number of classes and should be greater than 1
a blob with the correct class labels The dimensions should be the same as for the first input
Layer outputs:
the confusion matrix. The dimensions:
BatchLength, BatchWidth, ListSize, Depth, Channels are 1
Height and Width are equal to the input Channels

property
matrix
¶ Gets the confusion matrix. The dimensions:
BatchLength, BatchWidth, ListSize, Depth, Channels are 1
Height and Width are equal to the input Channels

property
reset
¶ Checks if the calculations will be reset on each run.

reset_matrix
()¶ Resets the confusion matrix values. The dimensions:
BatchLength, BatchWidth, ListSize, Depth, Channels are 1
Height and Width are equal to the input Channels
Working with discrete features¶
AccumulativeLookup¶

class
neoml.Dnn.
AccumulativeLookup
(input_layer, count, size, name=None)¶ The layer that trains fixedlength vector representations for the values of a discrete feature. It can work only with one feature. When several values of the feature are passed, the sum of the corresponding vectors is returned.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
count (int) – The number of vectors in the representation table.
size (int) – The length of each vector in the representation table.
name (str, default=None) – The layer name.
Layer inputs:
a data blob with integer data that contains the feature values. The dimensions:
BatchLength * BatchWidth * ListSize equal to the number of different values the feature can take
Height * Width * Depth * Channels equal to the number of values in the set
Layer outputs:
a blob with the sum of vector representations of the given feature values. The dimensions:
BatchLength, BatchWidth, ListSize equal to these dimensions of the input
Height, Width, Depth equal to 1
Channels equal to the vector length (size parameter below)

property
count
¶ Gets the number of vectors.

property
size
¶ Gets the vector length.
MultichannelLookup¶

class
neoml.Dnn.
MultichannelLookup
(input_layers, dimensions=None, name=None)¶ The layer that trains fixedlength vector representation for the values of several discrete features. See https://en.wikipedia.org/wiki/Word2vec, https://en.wikipedia.org/wiki/GloVe_(machine_learning)
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
dimensions (list of tuple(int, int)) – Each of the elements specifies the number and length of vectors in the representation table with the element’s index.
name (str, default=None) – The layer name.
Layer inputs:
a blob with feature values (float or int). The dimensions:
BatchLength * BatchWidth * ListSize * Height * Width * Depth is the number of features
Channels is the dimension along which the feature values for different sets are stored. Not smaller than the number of feature sets.
Layer outputs:
the blob with vectors for each input feature. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Depth are the same as for the input
Channels is the sum of vector lengths of all sets and additional channels if the input Channels is more than the number of tables.

property
dimensions
¶ Gets the list of representation table sizes.

get_embeddings
(index)¶ Gets the representation table with the given index as a blob of the dimensions:
BatchLength * BatchWidth * ListSize is dimensions[i].VectorCount
Height * Width * Depth * Channels is dimensions[i].VectorSize

initialize
(initializer)¶ Specifies a different initializer for this layer than the one set for the whole network in general.

set_embeddings
(index, blob)¶ Sets the representation table with the given index as a blob of the dimensions:
BatchLength * BatchWidth * ListSize is dimensions[i].VectorCount
Height * Width * Depth * Channels is dimensions[i].VectorSize
PositionalEmbedding¶

class
neoml.Dnn.
PositionalEmbedding
(input_layer, type_name, name=None)¶ The layer that maps positions in sequence into vectors, optionally trainable. The exact formula depends on the type_name parameter.
The formula for “transformers”: result[i][j][k] = input[i][j][k] + sin(j / pow(10000, (k / vectorLength))), where:
i is the index of sequence in batch (from 0 to BatchWidth  1)
j is the position of vector in sequence (from 0 to ListSize  1)
k is the index of the element in the vector (from 0 to vectorLength  1)
The formula for “learnable_addition”: result[i][j] = input[i][j] + addends[j], where:
i is the index of sequence in batch (from 0 to BatchWidth  1)
j is the position of vector in sequence (from 0 to ListSize  1)
addends is the trainable vector to be added
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
type_name (str, {"learnable_addition", "transformers"}) – The operation type.
name (str, default=None) – The layer name.
Layer inputs:
a blob with vector sequences. The dimensions:
BatchLength is 1
BatchWidth is the number of sequences in the set
ListSize is the sequence length
Height * Width * Depth * Channels is the vector length; for “transformers”, Height, Width, and Depth should be 1, vector length is equal to Channels
Layer outputs:
the transformation result, of the same dimensions as the input.

property
addends
¶ Gets the trainable vectors added. The blob dimensions:
BatchLength, BatchWidth are 1
the other dimensions are the same as for the input

property
type
¶ Gets the type of operation the layer performs.
TiedEmbeddings¶

class
neoml.Dnn.
TiedEmbeddings
(input_layers, embeddings_layer_name, channel, name=None)¶ The tied embeddings layer. See https://arxiv.org/pdf/1608.05859.pdf The representations table is taken from a MultichannelLookup layer.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
embeddings_layer_name (str) – The name of the layer used for embeddings. Needs to be a MultichannelLookup layer.
channel (int, >=0) – The channel index in the embeddings layer.
name (str, default=None) – The layer name.
Layer inputs:
The layer may have any number of inputs, of the dimensions:
BatchLength * BatchWidth * ListSize is the number of objects
Height, Width, Depth are 1
Channels is the embedding size
Layer outputs:
For each input the layer has one output of the same dimensions.

property
channel
¶ Gets the channel index in the embeddings layer.

property
embeddings_layer_name
¶ Gets the name of the layer used for representation table.
EnumBinarization¶

class
neoml.Dnn.
EnumBinarization
(input_layer, enum_size, name=None)¶ The layer that converts enumeration values into onehot encoding.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
enum_size (int, > 0) – The number of constants in the enumeration.
name (str, default=None) – The layer name.
Layer inputs:
a blob with int or float data that contains enumeration values. The dimensions:
Channels is 1
the other dimensions may be of any length
Layer outputs:
a blob with the vectors that onehot encode the enumeration values. The dimensions:
Channels is enum_size
the other dimensions stay the same as in the first input

property
enum_size
¶ Gets the number of constants in the enumeration.
BitSetVectorization¶

class
neoml.Dnn.
BitSetVectorization
(input_layer, bit_set_size, name=None)¶ The layer that converts a bitset into vectors of ones and zeros.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
bit_set_size (int, > 0) – The size of the bitset.
name (str, default=None) – The layer name.
Layer inputs:
a blob with int data containing bitsets. The dimensions:
BatchLength * BatchWidth * ListSize * Height * Width * Depth is the number of bitsets
Channels is bitset itself
Layer outputs:
a blob with the result of vectorization. The dimensions:
Channels is equal to bit_set_size
the other dimensions are the same as for the input

property
bit_set_size
¶ Gets the bitset size.
Attention layers¶
AttentionDecoder¶

class
neoml.Dnn.
AttentionDecoder
(input_layers, score, hidden_size, output_object_size, output_seq_len, name=None)¶ The layer that converts the input sequence into the output sequence, not necessarily of the same length
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
score (str, {'additive', 'dot_product'}) –
The type of estimate function used to check alignment of input and output sequences:
additive
is tanh(x*Wx + y*Wy)*vdot_product
is x*W*y
hidden_size (int) – The size of the hidden layer.
output_object_size (int) – The number of channels in the output object.
output_seq_len (int) – The length of the output sequence.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size with the input sequence
the special character used to initialize the output sequence All dimensions are equal to 1
Layer outputs:
the output sequence The dimensions:
BatchLength equal to output_seq_len
Channels equal to output_object_size
all other dimensions are equal to 1
Gets the size of the hidden layer.

property
output_object_size
¶ Gets the number of channels in the output.

property
output_seq_len
¶ Gets the length of the output sequence.

property
score
¶ Gets the estimate function.
MultiheadAttention¶

class
neoml.Dnn.
MultiheadAttention
(input_layer, head_count, hidden_size, output_size, dropout_rate, name=None)¶ The multihead selfattention layer that transforms a set of matrices according to a formula:
\(Q = W_Q * Q\), \(K = W_K * K\), \(V = W_V * V\) where \(W_*\) are trainable matrices of size (Channels_* x GetHiddenSize())
\(Attention(Q, K, V) = softmax( Q * K_t / \sqrt{d_K} ) * V\) where \(d_k\)  dimension of k
\(MultiHeadAttention = dropout\_if\_needed(concat( head_1, ..., head_N )) * W_O\) where \(head_i = Attention( W_{Q,i} * X, W_{K,i} * X, W_{V,i} * X )\), \(W_*\)  trainable parameters and \(W_O\) is an additional trainable matrix of size (GetHiddenSize() x GetOutputSize())
See the papers: https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1807.03819.pdf
 Parameters
input_layer (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
head_count (int, > 0, default=1) – The number of heads in attention.
hidden_size (int, > 0) – The size of trainable matrices W_*. Should be a multiple of head_count.
output_size (int, > 0) – The size of output.
dropout_rate (float, default=1) – Rate of droupout applied to the softmax. A negative value means no dropout.
name (str, default=None) – The layer name.
Layer inputs:
matrix Q of shape (1 x BatchWidth x ListSize_Q x 1 x 1 x 1 x Channels_Q)
matrix K of shape (1 x BatchWidth x ListSize_V x 1 x 1 x 1 x Channels_Q)
matrix V (1 x BatchWidth x ListSize_V x 1 x 1 x 1 x Channels_V)
(optional): mask, can be 0.
Layer outputs:
result matrix of shape (1, BatchWidth, ListSize_Q, 1, 1, 1, output_size)

property
dropout_rate
¶ Gets the rate of dropout applied to the softmax.

property
head_count
¶ Gets the number of heads in attention.
Gets the trainable matrices size.

property
output_size
¶ Gets the output size.

property
use_mask
¶ Checks if the mask will be used.
Logical operations Layers¶
Not¶

class
neoml.Dnn.
Not
(input_layer, name=None)¶ The layer that calculates logical not for each element of the single input:
\(f(x) = x == 0 ? 1 : 0\)
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a blob with integer data of any size
Layer outputs:
a blob with integer data of the same size with the result
Less¶

class
neoml.Dnn.
Less
(input_layers, name=None)¶ The layer that compares 2 inputs elementbyelement:
\(less[i] = first[i] < second[i] ? 1 : 0\)
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer must have 2 inputs of the same size and data type.
Layer outputs:
A blob with integer data of the same size as the inputs.
Equal¶

class
neoml.Dnn.
Equal
(input_layers, name=None)¶ The layer that compares 2 inputs elementbyelement:
\(equal[i] = first[i] == second[i] ? 1 : 0\)
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer must have 2 inputs of the same size and data type.
Layer outputs:
A blob with integer data of the same size as the inputs.
Where¶

class
neoml.Dnn.
Where
(input_layers, name=None)¶ The layer that merges 2 blobs based on a mask:
\(where[i] = first[i] != 0 ? second[i] : third[i]\)
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer must have 3 inputs of the same size. First input must be of integer data type. Data types of second and third inputs must match.
Layer outputs:
A blob with integer data of the same size and data type as the second input.
Auxiliary operations¶
Transform¶

class
neoml.Dnn.
Transform
(input_layer, transforms, name=None)¶ The layer that changes the input blob dimensions without moving any of the data. The total number of elements in the blob stays the same, and therefore the product of all dimensions should not be changed by the transformation.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
transforms (array of 7 tuples (operation, value), operation: one of "remainder", "set", "multiply", "divide" value: int > 0) –
Specifies the transformation to be made to each of the 7 dimensions:
”set” its length to the int value
”multiply” by the int value
”divide” by the int value
”remainder” may only be set for one dimension; it will be set so that the total size of the blob stays the same
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size.
Layer outputs:
a blob of the dimensions determined by the rules:
the dimensions in “set” mode will be equal to the specified value
the dimensions in “multiply” mode will be value times larger
the dimensions in “divide” mode will be value times smaller
the dimension in “remainder” mode will be such that the total size of the input and the output are the same

property
transforms
¶ Gets the array of transformations.
Transpose¶

class
neoml.Dnn.
Transpose
(input_layer, first_dim='height', second_dim='width', name=None)¶ The layer that switches two of the blob dimensions, moving the data inside accordingly.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
first_dim (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – One of the dimensions that should be switched.
second_dim (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – The other dimension that should be switched.
name (str, default=None) – The layer name.
Layer inputs:
a data blob to be transposed, of any size.
Layer outputs:
the result of transposition.

property
first_dim
¶ Gets the first dimension to be switched.

property
second_dim
¶ Gets the second dimension to be switched.
Argmax¶

class
neoml.Dnn.
Argmax
(input_layer, dimension='channels', name=None)¶ The layer that finds the maximum element along the given dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {'batch_length', 'batch_width', 'list_size', 'height', 'width', 'depth', 'channels'}, default='channels') – The dimension along which the maximum is to be found.
name (str, default=None) – The layer name.
Layer inputs:
a data blob of any size
Layer outputs:
contains the coordinate of the maximum The dimensions:
‘dimension’ along which the maximum is found is equal to 1
all other dimensions are equal to the first input dimensions

property
dimension
¶ Gets the dimension along which the maximum is to be found.
DotProduct¶

class
neoml.Dnn.
DotProduct
(input_layers, name=None)¶ The layer that calculates the dot product of its two inputs: each object in the first input is multiplied by the object with the same index in the second input.
 Parameters
input_layer (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer has two inputs, which must contain blobs of the same dimensions.
Layer outputs:
a blob with the dot product. The dimensions:
BatchLength, BatchWidth, ListSize equal to the inputs’ dimensions
Height, Width, Depth, Channels equal to 1
MatrixMultiplication¶

class
neoml.Dnn.
MatrixMultiplication
(input_layers, name=None)¶ The layer that multiplies two sets of matrices.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
the first set of matrices. The dimensions:
BatchLength * BatchWidth * ListSize  the number of matrices in the set
Height * Width * Depth  the height of each matrix
Channels  the width of each matrix
the second set of matrices. The dimensions:
BatchLength * BatchWidth * ListSize  the number of matrices in the set, must be the same as for the first input
Height * Width * Depth  the height of each matrix, must be equal to Channels of the first input
Channels  the width of each matrix
Layer outputs:
the set of multiplication results. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Depth the same as for the first input
Channels the same as for the second input
Reorg¶

class
neoml.Dnn.
Reorg
(input_layer, stride=1, name=None)¶ The layer that transforms a set of twodimensional multichannel images into a set of images of smaller size but with more channels. This operation is used in YOLO architecture: https://pjreddie.com/darknet/yolo/
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
stride (int, default=1) – The value by which the image size will be divided in the final result.
name (str, default=None) – The layer name.
Layer inputs:
a blob with images. The dimensions:
BatchLength * BatchWidth * ListSize is the number of images
Height is the image height; should be a multiple of stride
Width is the image width; should be a multiple of stride
Depth is 1
Channels is the number of channels the image format uses
Layer outputs:
the result of image transformation. The dimensions:
BatchLength, BatchWidth, ListSize are the same as for the input
Height is input Height / stride
Width is input Width / stride
Depth is 1
Channels is input Channels * stride^2

property
stride
¶ Gets the divider value.
Concatenation layers¶
The layers that concatenate blobs along one of the dimensions.
ConcatChannels¶

class
neoml.Dnn.
ConcatChannels
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the Channels dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Depth equal for all inputs
Channels dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Depth equal to the inputs’ dimensions
Channels equal to the sum of all inputs’ Channels
ConcatDepth¶

class
neoml.Dnn.
ConcatDepth
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the Depth dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Channels equal for all inputs
Depth dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Width, Channels equal to the inputs’ dimensions
Depth equal to the sum of all inputs’ Depth
ConcatWidth¶

class
neoml.Dnn.
ConcatWidth
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the Width dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Depth, Channels equal for all inputs
Width dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, ListSize, Height, Depth, Channels equal to the inputs’ dimensions
Width equal to the sum of all inputs’ Width
ConcatHeight¶

class
neoml.Dnn.
ConcatHeight
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the Height dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, ListSize, Width, Depth, Channels equal for all inputs
Height dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, ListSize, Width, Depth, Channels equal to the inputs’ dimensions
Height equal to the sum of all inputs’ Height
ConcatBatchWidth¶

class
neoml.Dnn.
ConcatBatchWidth
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the BatchWidth dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, ListSize, Height, Width, Depth, Channels equal for all inputs
BatchWidth dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, ListSize, Height, Width, Depth, Channels equal to the inputs’ dimensions
BatchWidth equal to the sum of all inputs’ BatchWidth
ConcatBatchLength¶

class
neoml.Dnn.
ConcatBatchLength
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the BatchLength dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchWidth, ListSize, Height, Width, Depth, Channels equal for all inputs
BatchLength dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchWidth, ListSize, Height, Width, Depth, Channels equal to the inputs’ dimensions
BatchLength equal to the sum of all inputs’ BatchLength
ConcatListSize¶

class
neoml.Dnn.
ConcatListSize
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the ListSize dimension.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, Height, Width, Depth, Channels equal for all inputs
ListSize dimension may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, Height, Width, Depth, Channels equal to the inputs’ dimensions
ListSize equal to the sum of all inputs’ ListSize
ConcatObject¶

class
neoml.Dnn.
ConcatObject
(input_layers, name=None)¶ The layer that concatenates several blobs into one along the Height, Width, Depth, and Channels dimensions.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer accepts an arbitrary number of inputs. The dimensions:
BatchLength, BatchWidth, ListSize equal for all inputs
Height, Width, Depth, Channels dimensions may vary
Layer outputs:
a blob with the result of concatenation. The dimensions:
BatchLength, BatchWidth, ListSize equal to the inputs’ dimensions
Height, Width, Depth equal to 1
Channels equal to the sum of Height * Width * Depth * Channels over all inputs
Split layers¶
The layers that split a blob along one of the dimensions.
SplitChannels¶

class
neoml.Dnn.
SplitChannels
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the Channels dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
Channels should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
Channels equals the corresponding element of sizes array, for the last output it is input Channels minus the sum of sizes
all other dimensions are the same as for the input
SplitDepth¶

class
neoml.Dnn.
SplitDepth
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the Depth dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
Depth should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
Depth equals the corresponding element of sizes array, for the last output it is input Depth minus the sum of sizes
all other dimensions are the same as for the input
SplitWidth¶

class
neoml.Dnn.
SplitWidth
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the Width dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
Width should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
Width equals the corresponding element of sizes array, for the last output it is input Width minus the sum of sizes
all other dimensions are the same as for the input
SplitHeight¶

class
neoml.Dnn.
SplitHeight
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the Height dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
Height should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
Height equals the corresponding element of sizes array, for the last output it is input Height minus the sum of sizes
all other dimensions are the same as for the input
SplitListSize¶

class
neoml.Dnn.
SplitListSize
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the ListSize dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
ListSize should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
ListSize equals the corresponding element of sizes array, for the last output it is input ListSize minus the sum of sizes
all other dimensions are the same as for the input
SplitBatchWidth¶

class
neoml.Dnn.
SplitBatchWidth
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the BatchWidth dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
BatchWidth should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
BatchWidth equals the corresponding element of sizes array, for the last output it is input BatchWidth minus the sum of sizes
all other dimensions are the same as for the input
SplitBatchLength¶

class
neoml.Dnn.
SplitBatchLength
(input_layer, sizes, name=None)¶ The layer that splits an input blob along the BatchLength dimension.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.
Layer inputs:
a blob with input data. The dimensions:
BatchLength should not be less than the sum of sizes array elements.
Layer outputs:
The layer has at least len(sizes) outputs. The dimensions:
BatchLength equals the corresponding element of sizes array, for the last output it is input BatchLength minus the sum of sizes
all other dimensions are the same as for the input
Working with sequences¶
SubSequence¶

class
neoml.Dnn.
SubSequence
(input_layer, start_pos=0, length=1, name=None)¶ The layer that extracts a subsequence from each vector sequence of the set.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
start_pos (int) – The first element of the subsequence. Counted from the end if negative.
length (int) – The length of the subsequence. Reversed order if negative.
name (str, default=None) – The layer name.
Layer inputs:
a blob with a set of objects, numbered along the BatchWidth dimension.
Layer outputs:
a blob with the subsequence of objects. The dimensions:
BatchWidth is abs(length) or smaller if it doesn’t fit after starting at start_pos
the other dimensions are the same as for the input

property
length
¶ Gets the subsequence length.

property
start_pos
¶ Gets the starting position.
ReverseSequence¶

class
neoml.Dnn.
ReverseSequence
(input_layer, name=None)¶ The layer that reverses sequence order of the input.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a blob of any size with sequence of objects numbered along the BatchWidth dimension.
Layer outputs:
the reverse sequence. The same size as the input.
SequenceSum¶

class
neoml.Dnn.
SequenceSum
(input_layer, name=None)¶ The layer that adds up object sequences.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a set of object sequences. The dimensions:
BatchLength is the sequence length
BatchWidth is the number of sequences in the set
ListSize is 1
Height * Width * Depth * Channels is the object size
Layer outputs:
the results of adding up each of the sequences. The dimensions:
BatchLength is 1
the other dimensions are the same as for the input
Image conversion layers¶
ImageResize¶

class
neoml.Dnn.
ImageResize
(input_layer, deltas, default_value=0.0, name=None)¶ The layer that resizes a set of twodimensional multichannel images.
 Parameters
input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
deltas (str, ("left", "right", "top", "bottom")) – The differences between the original and the resized image, on each side. If the difference is negative, rows or columns are removed from the specified side. If it is positive, rows or columns are added and filled with the default_value pixels.
default_value (float, default=0.0) – The value for the added pixels.
name (str, default=None) – The layer name.
Layer inputs:
a set of images, of the dimensions:
BatchLength * BatchWidth * ListSize  the number of images
Height  the images’ height
Width  the images’ width
Depth * Channels  the number of channels the image format uses
Layer outputs:
a blob with the resized images, of the dimensions:
BatchLength, BatchWidth, ListSize, Depth, Channels are equal to the input dimensions
Height is the input Height plus the sum of top and bottom deltas
Width is the input Width plus the sum of right and left deltas

property
default_value
¶ Gets the default value for new pixels.

property
deltas
¶ Gets the size differences on each side.
PixelToImage¶

class
neoml.Dnn.
PixelToImage
(input_layer, height, width, name=None)¶ The layer that creates a set of twodimensional images using a set of pixel sequences with specified coordinates.
 Parameters
input_layer (list of object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
height (int) – The height of the resulting images.
width (int) – The width of the resulting images.
name (str, default=None) – The layer name.
Layer inputs:
a blob with pixel sequences. The dimensions:
BatchLength is 1
BatchWidth is the number of sequences in the set
ListSize is the length of each sequence
Height, Width, Depth are 1
Channels is the number of channels for the pixel sequences and the output images.
a blob with integer data that contains lists of pixel coordinates. The dimensions:
BatchWidth, ListSize are the same as for the first input
the other dimensions are 1
Layer outputs:
a blob with images. The dimensions:
BatchLength is 1
BatchWidth is the same as for the first input
ListSize is 1
Height is the specified image height
Width is the specified image width
Depth is 1
Channels is the same as for the first input

property
height
¶ Gets the output image height.

property
width
¶ Gets the output image width.
ImageToPixel¶

class
neoml.Dnn.
ImageToPixel
(input_layer, name=None)¶ The layer that extracts a set of pixel sequences along the specified coordinates from a set of twodimensional images.
 Parameters
input_layer (list of object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.
Layer inputs:
a set of twodimensional images. The blob dimensions:
BatchLength is 1
BatchWidth is the number of sequences in the set
ListSize 1
Height is the images’ height
Width is the images’ width
Depth is 1
Channels is the number of channels the image format uses
a blob with integer data that contains the pixel sequences. The dimensions:
BatchWidth is the same as for the first input
ListSize is the length of each sequence
all other dimensions are 1
Layer outputs:
a blob with the pixel sequences. The dimensions:
BatchLength is 1
BatchWidth is the inputs’ BatchWidth
ListSize is the same as for the second input
Height, Width, Depth are 1
Channels is the same as for the first input
Upsampling2D¶

class
neoml.Dnn.
Upsampling2D
(input_layers, height_copy_count, width_copy_count, name=None)¶ The layer that scales up a set of twodimensional multichannel images. The new pixels are filled up by repeating the existing pixels’ values, without any interpolation.
 Parameters
input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
height_copy_count (int, > 0) – The height multiplier.
width_copy_count (int, > 0) – The width multiplier.
name (str, default=None) – The layer name.
Layer inputs:
The layer can have any amount of inputs, each with a set of images. The dimensions:
BatchLength * BatchWidth * ListSize is the number of images
Height is the images’ height
Width is the images’ width
Depth * Channels is the number of channels the image format uses
Layer outputs:
For each input, there is a corresponding output with upscaled images. The dimensions:
BatchLength, BatchWidth, ListSize, Depth, Channels equal the correspoonding input dimensions
Height is height_copy_count times larger than the input Height
Width is width_copy_count times larger than the input Width

property
height_copy_count
¶ Gets the height multiplier.

property
width_copy_count
¶ Gets the width multiplier.
Scatter & Gather Layers¶
ScatterND¶

class
neoml.Dnn.
ScatterND
(input_layers, name=None)¶ The layer that updates certain objects in data:
\(data[indices[i]] = updates[i]\)
where indices[…] is an integer vector of IndexDims elements containing coordinates in the first IndexDims dimensions of the data blob
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
The layer must have 3 inputs.
1. Data. Set of objects of any type. The product of its first IndexDims dimensions is ObjectCount. The product of the rest of the dimensions is ObjectSize.
2. Indices. Blob of integer data type. The number of channels of this blob is IndexDims. The total size must be UpdateCount * IndexDims.
3. Updates. Blob of the same data type as the first. The total size of the blob is UpdateCount * ObjectSize.
Layer outputs:
A blob of the same data type and size as first input.
Conditional random field (CRF)¶
Conditional random field is implemented in three parts:
the trainable layer that contains the transition probabilities of the random field
the loss function optimized by training
the special layer to extract highestprobability sequences from the output of the trained CRF layer
Crf¶

class
neoml.Dnn.
Crf
(input_layer, class_count, padding=0, dropout_rate=0.0, name=None)¶ The layer that trains and calculates transitional probabilities in a conditional random field (CRF).
 Parameters
input_layer (object, tuple(object, int) of list of them) – The input layer and the number of its output. If no number is specified, the first output will be connected.
class_count (int) – The number of classes in the CRF.
padding (int, default=0) – The number of empty class used to fill the sequence end.
dropout_rate (float, default=0.0) – Variational dropout.
name (str, default=None) – The layer name.
Layer inputs:
a blob with the object sequences. The dimensions:
BatchLength is the sequence length
BatchWidth is the number of sequences in the set
ListSize is 1
Height * Width * Depth * Channels is the object size
(optional): a blob with integer data that contains the correct class sequences. It is required only for training. The dimensions:
BatchLength, BatchWidth equal to the first input’s
the other dimensions are equal to 1
Layer outputs:
(optional): a blob with integer data that contains optimal class sequences. This output will be returned only if you set calc_prev_best_class to True. During training, this output usually isn’t needed and is switched off by default. The dimensions:
BatchLength, BatchWidth equal to the first input’s
Channels equal to the number of classes
the other dimensions are equal to 1
a blob with float data that contains nonnormalized logarithm of optimal class sequences probabilities. The dimensions are the same as for the first output.
(optional): a blob with nonnormalized logarithm of the correct class sequences probabilities. This output will be there only if the layer has two inputs. The dimensions are equal to the second input’s:
BatchLength, BatchWidth equal to the first input’s
the other dimensions are equal to 1

property
calc_best_prev_class
¶ Checks if the first output will be returned.

property
class_count
¶ Gets the number of classes in the CRF.

property
dropout_rate
¶ Gets the variational dropout rate.

property
free_terms
¶ Gets the hidden layer free terms. The blob size is class_count.
Gets the hidden layer weights. The dimensions:
BatchLength * BatchWidth * ListSize is the number of classes
Height * Width * Depth * Channels the same as for the first input

property
padding
¶ Gets the number of the empty class.

property
transitions
¶ Gets the transition probability matrix. The dimensions:
BatchLength, BatchWidth are class_count
the other dimensions are 1
CrfLoss¶

class
neoml.Dnn.
CrfLoss
(input_layers, loss_weight=1.0, name=None)¶ The layer that calculates the loss function used for training a CRF. The value is log(probability of the correct class sequence)
 Parameters
input_layers (array of (object, int) tuples or objects) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
a blob with integer data that contains optimal class sequences. The dimensions:  BatchLength, BatchWidth equal to the network inputs’  Channels equal to the number of classes  the other dimensions are equal to 1
a blob with float data containing nonnormalized logarithm of probabilities of the optimal class sequences. The dimensions are the same as for the first input.
a blob with float data containing nonnormalized logarithm of probability of the correct class being in this position. The dimensions:
BatchLength, BatchWidth the same as for the first input
the other dimensions equal to 1
Layer outputs:
The layer has no output.

property
last_loss
¶ Gets the value of the loss function on the last step.

property
loss_weight
¶ Gets the multiplier for the loss function value dduring training.

property
max_gradient
¶ Gets the gradient clipping threshold.
BestSequence¶

class
neoml.Dnn.
BestSequence
(input_layers, name=None)¶ The layer that finds the optimal class sequence using the Crf layer output.
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.
Layer inputs:
first output of Crf. A blob with int data that contains the optimal class sequences. The dimensions:
BatchLength is the sequence length
BatchWidth is the number of sequences in the set
Channels is the number of classes
all other dimensions are 1
second output of Crf. A blob with float data that contains nonnormalized logarithm of optimal class sequences probabilities. The dimensions are the same as for the first input.
Layer outputs:
a blob with int data that contains the optimal class sequence. The dimensions:
BatchLength, BatchWidth are the same as for the inputs
the other dimensions are 1
Connectionist temporal classification (CTC)¶
A connectionist temporal classification (CTC) network is trained to optimize the loss function.
After training, use the special decoding layer to extract the optimal sequences from the network output.
See also https://www.cs.toronto.edu/~graves/preprint.pdf
CtcLoss¶

class
neoml.Dnn.
CtcLoss
(input_layers, blank, skip, loss_weight=1.0, name=None)¶ The layer that calculates the loss function used for connectionist temporal classification (CTC). See https://www.cs.toronto.edu/~graves/preprint.pdf
 Parameters
input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
blank (int) – Sets the value for the blank label that will be used as space.
skip (bool) – Specifies if blank labels may be skipped when aligning.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.
Layer inputs:
the network response. The dimensions:
BatchLength is the maximum sequence length
BatchWidth is the number of sequences in the set
ListSize is 1
Height * Width * Depth * Channels is the number of classes
the correct labels as a blob with int data.
The dimensions:  BatchLength is the maximum labels sequence length  BatchWidth is the number of sequences, same as first input’s BatchWidth  the other dimensions are 1
(optional): the label sequences lengths as a blob with int data. If this input isn’t connected, the label sequences are considered to be the second input’s BatchLength long. The dimensions:
BatchWidth is the same as for the first input
the other dimensions are 1
(optional): the network response sequences lengths as a blob with int data. If this input isn’t connected, the response sequences are considered to be the first input’s BatchLength long. The dimensions:
BatchWidth is the same as for the first input
the other dimensions are 1
(optional): the sequences’ weights. The dimensions:
BatchWidth is the same as for the first input
the other dimensions are 1
Layer outputs:
The layer has no output.

property
blank
¶ Gets the value of the blank label.

property
last_loss
¶ Gets the value of the loss function on the last step.

property
loss_weight
¶ Gets the multiplier for the loss function during training.

property
max_gradient
¶ Gets the upper limit for the absolute value of the function gradient.

property
skip
¶ Checks if blank labels may be skipped when aligning.
CtcDecoding¶

class
neoml.Dnn.
CtcDecoding
(input_layers, blank, blank_threshold, arc_threshold, name=None)¶ The layer that is looking for the most probable sequences in the response of a connectionist temporal classification (CTC) network.
 Parameters
input_layers (object, (object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
blank (int) – Sets the value for the blank label that will be used as space.
blank_threshold (float, [0..1]) – The probability threshold for blank labels when building a linear division graph (LDG).
arc_threshold (float, [0..1]) – The probability threshold for cutting off arcs when building a linear division graph (LDG).
name (str, default=None) – The layer name.
Layer inputs:
the network response. The dimensions:
BatchLength is the maximum sequence length
BatchWidth is the number of sequences in the set
ListSize is 1
Height * Width * Depth * Channels is the number of classes
(optional): the network response sequences lengths as a blob with int data. If this input isn’t connected, the response sequences are considered to be the first input’s BatchLength long. The dimensions:
BatchWidth is the same as for the first input
the other dimensions are 1
Layer outputs:
The layer has no output.

property
arc_threshold
¶ Gets the probability threshold for cutting off arcs when building a linear division graph (LDG).

property
batch_width
¶ Returns the number of sequences.

property
blank
¶ Gets the value of the blank label.

property
blank_threshold
¶ Gets the probability threshold for blank layers when building a linear division graph (LDG).

get_best_sequence
(sequence_number)¶ Retrieves the most probable sequence for the object with sequence_number index in the set.

property
label_count
¶ Returns the number of classes.

property
sequence_length
¶ Returns the sequence length.
Initializers¶
Before the first training iteration the layers’ weights (trainable parameters) must be initialized. The initializer is the same for all the network trainable weights, except for the free term vectors that are initialized with zeros.
Xavier¶
The default initializer. It generates the weights using the normal distribution N(0, 1/n)
, where n is the input size.

class
neoml.Dnn.
Xavier
(random_generator=None)¶ Initializes a blob using the Xavier algorithm: with random values from a normal distribution over (0, 1 / input_size).
 Parameters
random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.
XavierUniform¶
The default initializer. It generates the weights using the uniform distribution U(sqrt(1/n), sqrt(1/n))
, where n is the input size.

class
neoml.Dnn.
XavierUniform
(random_generator=None)¶ Initializes a blob using the Xavier algorithm: with random values from a uniform distribution over (sqrt(1 / input_size), sqrt(1 / input_size)).
 Parameters
random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.
Uniform¶
Generates the weights using a uniform distribution over the specified segment.

class
neoml.Dnn.
Uniform
(lower_bound=1.0, upper_bound=1.0, random_generator=None)¶ Initializes a blob using uniform distribution between the set bounds.
 Parameters
lower_bound (float, default=1.0) – The lower bound of the distribution.
upper_bound (float, default=1.0) – The upper bound of the distribution.
random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.

property
lower_bound
¶ Gets the distribution lower bound.

property
upper_bound
¶ Gets the distibution upper bound.
Solvers¶
The optimizer, or solver, sets the rules to update the weights during training.
SimpleGradient¶

class
neoml.Dnn.
SimpleGradient
(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=1.0, moment_decay_rate=0.9)¶ Stochastic gradient descent with moment.
 Parameters
math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.

property
moment_decay_rate
¶ Gets the moment decay rate. Moment is a weighted sum of previous gradients.
AdaptiveGradient¶

class
neoml.Dnn.
AdaptiveGradient
(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=1.0, moment_decay_rate=0.9, second_moment_decay_rate=0.99, epsilon=1e06, ams_grad=False, decoupled_weight_decay=False)¶ Gradient descent with adaptive momentum (Adam).
 Parameters
math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.
second_moment_decay_rate (float, default=0.99) – The decay rate for the weighted sum of previous gradients, squared, also called the second moment.
epsilon (float, default=1e6) – The small value used to avoid division by zero when calculating second moment.
ams_grad (bool, default=False) – Turns AMSGrad mode on or off. AMSGrad helps against divergence and rapid vanishing of previous states memory, which may become a problem for the optimizers that use the moving mean for squared gradient history (Adam, NAdam, RMSprop). See https://openreview.net/pdf?id=ryQu7fRZ.
decoupled_weight_decay (bool, default=False) – Enables using of Decoupled Weight Decay Regularization. See https://openreview.net/pdf?id=Bkg6RiCqY7.

property
ams_grad
¶ Checks if AMSGrad mode is on.

property
decoupled_weight_decay
¶ Checks if Decoupled Weight Decay Regularization is used.

property
epsilon
¶ Gets the small value used to avoid division by zero when calculating second moment.

property
moment_decay_rate
¶ Gets the moment decay rate. Moment is a weighted sum of previous gradients.

property
second_moment_decay_rate
¶ Gets the decay rate for the weighted sum of previous gradients, squared  that is, the second moment.
NesterovGradient¶

class
neoml.Dnn.
NesterovGradient
(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=1.0, moment_decay_rate=0.9, second_moment_decay_rate=0.99, epsilon=1e06, ams_grad=False, decoupled_weight_decay=False)¶ The optimizer that uses Nesterov moment. See http://cs229.stanford.edu/proj2015/054_report.pdf (Algo 8).
 Parameters
math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.
second_moment_decay_rate (float, default=0.99) – The decay rate for the weighted sum of previous gradients, squared, also called the second moment.
epsilon (float, default=1e6) – The small value used to avoid division by zero when calculating second moment.
ams_grad (bool, default=False) – Turns AMSGrad mode on or off. AMSGrad helps against divergence and rapid vanishing of previous states memory, which may become a problem for the optimizers that use the moving mean for squared gradient history (Adam, NAdam, RMSprop). See https://openreview.net/pdf?id=ryQu7fRZ.
decoupled_weight_decay (bool, default=False) – Enables using of Decoupled Weight Decay Regularization. See https://openreview.net/pdf?id=Bkg6RiCqY7.

property
ams_grad
¶ Checks if AMSGrad mode is on.

property
decoupled_weight_decay
¶ Checks if Decoupled Weight Decay Regularization is used.

property
epsilon
¶ Gets the small value used to avoid division by zero when calculating second moment.

property
moment_decay_rate
¶ Gets the moment decay rate. Moment is a weighted sum of previous gradients.

property
second_moment_decay_rate
¶ Gets the decay rate for the weighted sum of previous gradients, squared  that is, the second moment.
Random¶

class
neoml.Random.
Random
(seed=None)¶ The random numbers generator.
 Parameters
seed (int, default=None) – the seed of the random numbers generator.
Autodifferentiation¶
NeoML supports autodifferentiation for a wide set of operations. Use these operations and simple arithmetic if you’d like to create your own loss function neoml.Dnn.CustomLoss. Then during the backward pass, NeoML will be able to calculate gradients of your custom loss.

AutoDiff.
const
(shape, data)¶ Creates a blob of the specified shape filled with data value.
Simple arithmetic operations¶

AutoDiff.
add
(b)¶ Elementwise adds two blobs or a blob and a scalar value.

AutoDiff.
sub
(b)¶ Elementwise subtracts two blobs or a blob and a scalar value.

AutoDiff.
mul
(b)¶ Elementwise multiplies two blobs or a blob and a scalar value.

AutoDiff.
div
(b)¶ Elementwise divides two blobs or a blob and a scalar value.

AutoDiff.
less
(b)¶ Compare blobs elementwise.
Basic math functions¶

AutoDiff.
max
(b)¶ Takes the elementwise maximum of a blob and a scalar value.

AutoDiff.
sum
(axes=None)¶ Calculates sum of blob elements along provided axes. If axes=None calculates the total sum.

AutoDiff.
cumsum
(axis=0)¶ Calculates cumulative sum of blob elements along provided axis.

AutoDiff.
mean
(axes=None)¶ Calculates mean of blob elements along provided axes. If axes=None calculates the total mean.

AutoDiff.
neg
()¶ Returns the negative of a blob or a number.

AutoDiff.
abs
()¶ Takes absolute value of each blob element.

AutoDiff.
log
()¶ Takes the logarithm of each blob element.

AutoDiff.
exp
()¶ Takes the exponential of each blob element.

AutoDiff.
pow
(b)¶ Computes the power of one blob to another elementwise.
Other operations¶

AutoDiff.
concat
(axis=0)¶ Merges the blobs along given axis.

AutoDiff.
reshape
(shape)¶ Reshape the blob.

AutoDiff.
clip
(min_value, max_value)¶ Clips each element of the blob so that it fits between the specified limits.

AutoDiff.
top_k
(k=1)¶ Finds values of the k largest elements in the blob. The result is a blob of size k.

AutoDiff.
binary_cross_entropy
(preds, fromLogits)¶ Calculates binary crossentropy for two blobs: the first one contains labels, the second one contains predictions. Blobs should be of the same shape. \(result = (1  labels) * x + log(1 + exp(x))\) if fromLogits then x = preds, else \(x = log( clippedPreds / (1  clippedPreds))\)