neoml.Dnn

Neural network
Data blobs
Layers
Initializers
Solvers
Random
Autodifferentiation

Neural network

A neural network is a directed graph with the vertices corresponding to layers and the arcs corresponding to the connections along which the data is passed from one layer’s output to another’s input.

Each layer should be added to the network after you assign a unique name to it. A layer may not be connected to several networks at once.

Source layers are used to pass the data into the network. A source layer has no inputs and passes the data blob specified by the user to its only output.

Sink layers with no outputs are used to retrieve the result of the network operation. They provide a function that returns the blob with data.

After all the layers are added and connected the network may be set up for training.

class neoml.Dnn.Dnn(*args: Any, **kwargs: Any)

Neural network implementation. A neural network is a directed graph consisting of layers that perform calculations on data blobs. It starts with source layers and ends with sink layers.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – The math engine that will perform calculations.
random (object, default=None) – The random numbers generator to be used for training and initialization.

add_layer(layer)

Adds a layer to the network.

Parameters:: layer (neoml.Dnn.Layer) – the layer to be added.

delete_layer(layer)

Deletes a layer from the network.

Parameters:: layer (neoml.Dnn.Layer or str) – the layer to be deleted, or its name

property initializer: The initializer that will fill in the weight values before training starts. Xavier initialization is the default.

property input_layers: All source layers of the network.

property layers: Gets all layers of the network.

learn(inputs)

Runs the network, performs a backward pass and updates the trainable weights.

Parameters:: inputs (dict) – The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.

load(path)

Loads the network from file.

Parameters:: path (str) – The full path to the location from where the network should be loaded.

load_checkpoint(path)

Loads the checkpoint from file. A new solver will be created, because the old pointers will point to an object no longer used by this network.

Parameters:: path (str) – The full path to the location from where the network should be loaded.

property math_engine: The math engine neoml.MathEngine.MathEngine used by the network.

property output_layers: All sink layers of the network.

run(inputs)

Runs the network.

Parameters:: inputs – The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.
Type:: inputs: dict

run_and_backward(inputs)

Runs the network and performs a backward pass with the input data.

Param:: inputs: The dictionary of input blobs. The dictionary keys (str) are the source layer names. The dictionary values (neoml.Blob.Blob) are the blobs passed to these source layers.

property solver: The optimizer for the layer’s trainable parameters.

store(path)

Serializes the network.

Parameters:: path (str) – The full path to the location where the network should be stored.

store_checkpoint(path)

Serializes the network with the data required to resume training.

Parameters:: path (str) – The full path to the location where the network should be stored.

Data blobs

All data used in the network operation (inputs, outputs, trainable parameters) is stored in blobs. A blob is a 7-dimensional array, and each of its dimensions has a specific meaning:

BatchLength is a “time” axis, used to denote data sequences; it is mainly used in recurrent networks
BatchWidth corresponds to the batch, used to pass several independent objects together
ListSize is the dimensions for the objects that are connected (for example, pixels out of one image) but do not form a sequence
Height is the height of a matrix or an image
Width is the width of a matrix or an image
Depth is the width of a 3-dimensional image
Channels corresponds to channels for multi-channel image formats and is also used to work with one-dimensional vectors.

The blobs may contain one of the two types of data: float and int. Both data types are 32-bit.

If the data type is not specified directly anywhere in this documentation, that means float is used.

Class description

class neoml.Blob.Blob(internal)

The class that stores and transmits data in neural networks.

asarray(copy=False, keep_dims=False)

Returns the contents of the blob as a multi-dimensional array.

Parameters:

copy (bool, default=False) – if True, the data will be copied. If False, the array may share the memory buffer with the blob if possible and only provide more convenient access to the same data. Not copying may be impossible if the blob is in GPU memory.
keep_dims – if True then the result shape will contain all the 7

dimensions of the blob. Otherwise it will contain only dimensions longer than 1. If False and blob consists only of one element, then the result will be of shape (1,). :type keep_dims: bool, default=False

property batch_len: The BatchLength dimension.

property batch_width: The BatchWidth dimension.

property channels: The Channels dimension.

copy(math_engine): Creates a blob copy independent of this blob.

property depth: The Depth dimension.

property height: The Height dimension.

property list_size: The ListSize dimension.

property math_engine: The math engine that allocates the blob memory and will work with its data.

property object_count: The number of objects in the blob, equal to BatchLength * BatchWidth * ListSize.

property object_size: The size of one object, equal to Height * Width * Depth * Channels.

property shape: The blob shape: the array of 7 int elements with dimension lengths.

property size: The blob total size: the product of all dimensions.

property width: The Width dimension.

Working with blobs

Blob.store(file_path)

Stores the blob in a file at the specified path.

Parameters:: file_path (str) – the full path to the file where the blob should be stored.

Blob.load(file_path)

Loads the blob from the specified location.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
file_path (str) – the full path to the file from which the blob should be loaded.

Blob.asblob(data, shape=None, copy=False)

Organizes the data from a memory buffer into a blob.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
data (object) – a pointer to the data.
shape (array of int, default=None) – the target blob dimensions. If none are specified, a one-element blob will be assumed.
copy (bool, default=False) – specifies if the data should be copied to another memory block or kept in the same place if possible, making the data parameter point to the body of the newly-created blob. Not copying may be impossible if the blob is in GPU memory.

Creating blobs of typical size

The auxiliary methods that create blobs for frequently used types of data.

Blob.vector(size, dtype='float32')

Creates a one-dimensional blob, that is, a vector.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
size (int, > 0) – the vector length.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.matrix(matrix_height, matrix_width, dtype='float32')

Creates a two-dimensional blob, that is, a matrix.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
matrix_height (int, > 0) – the matrix height.
matrix_width (int, > 0) – the matrix width.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.tensor(shape, dtype='float32')

Creates a blob of the specified shape.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
shape (array of int) – the target blob dimensions.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.list_blob(batch_len, batch_width, list_size, channels, dtype='float32')

Creates a blob with one-dimensional Height * Width * Depth elements.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
list_size (int, > 0) – the ListSize dimension of the new blob.
channels (int, > 0) – the Channels dimension of the new blob.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.image2d(batch_len, batch_width, height, width, channels, dtype='float32')

Creates a blob with two-dimensional multi-channel images.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
height (int, > 0) – the image height.
width (int, > 0) – the image width.
channels (int, > 0) – the number of channels in the image format.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Blob.image3d(batch_len, batch_width, height, width, depth, channels, dtype='float32')

Creates a blob with three-dimensional multi-channel images.

Parameters:

math_engine (neoml.MathEngine.MathEngine) – the math engine that works with this blob.
batch_len (int, > 0) – the BatchLength dimension of the new blob.
batch_width (int, > 0) – the BatchWidth dimension of the new blob.
height (int, > 0) – the image height.
width (int, > 0) – the image width.
depth (int, > 0) – the image depth.
channels (int, > 0) – the number of channels in the image format.
dtype (str, {"float32", "int32"}, default="float32") – the type of data in the blob.

Layers

A layer is an element of the network that performs some operation: anything from the input data reshape or a simple math function calculation, up to convolution or LSTM (Long short-term memory).

If the operation needs input data, it will be taken from the layer input. Each layer input contains one data blob, and if several blobs are needed, the layer will have several inputs. Each layer input should be connected to another layer’s output.

If the operation returns results that should be used by other layers, they will be passed to the layer outputs. Each layer output contains one data blob, so depending on the operation it performs the layer may have several outputs. Several other layer inputs may be connected to the same output, but you may not leave an output unconnected to any inputs.

In addition, the layer may have settings specified by the user before starting calculations, and trainable parameters that are optimized during network training.

The layers also have names that can be used to find a layer in the network. The name should be set at layer creation or before adding it to the network.

Base layer class

All NeoML layer classes are derived from this class.

class neoml.Dnn.Layer(internal)

The base class for a network layer.

property class_name: String containing NeoML class of the layer

connect(layer, output_index=0, input_index=0)

Connects this layer to another.

Parameters:

layer (neoml.Dnn.Layer) – the layer to which this one will be connected.
output_index (int, >=0, default=0) – the number of the layer’s output to be connected.
input_index (int, >=0, default=0) – the number of this layer’s input to be connected.

property input_links: Tuple in which i’th element contains complete information about the link between i’th input layer and self (layer name, output idx)

property input_names: Tuple in which i’th element contains the name of the layer, connected to the i’th inputs of self

property learning_enabled: Gets whether the weights are frozen or not.

property name: Gets the layer name.

Passing the data to and from the network

Source

class neoml.Dnn.Source(dnn, name=None)

The source layer that serves to pass a user data blob into the network.

Parameters:

dnn (object) – The neural network.
name (str, default=None) – The layer name.

Layer inputs:

The layer has no inputs.

Layer outputs:

The layer has one output that contains the data blob passed into the last call of set_blob.

get_blob(): Gets the blob with source data.

set_blob(blob): Sets the blob with source data.

Sink

class neoml.Dnn.Sink(input_layer, name=None)

The sink layer that serves to pass a data blob out of the network. Use the get_blob method to retrieve the blob.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

The layer has one input that accepts a data blob of any size.

Layer outputs:

The layer has no outputs.

get_blob(): Gets the data blob.

Recurrent layers

Lstm

The long short-term memory layer.

class neoml.Dnn.Lstm(input_layer, hidden_size=1, dropout_rate=0.0, activation='sigmoid', reverse_seq=False, name=None)

A long short-term memory (LSTM) layer that can be applied to a set of vector sequences. The output is a sequence containing the same number of vectors, each of the layer hidden size length.

Parameters:

input_layer (object, tuple(object, int) or list of them) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size and the LSTM state vector size.
dropout_rate (float, default=0.0) – The dropout probability for the input and the recurrent data.
recurrent_activation (str, {"linear", "elu", "relu", "leaky_relu", "abs", "sigmoid", "tanh", "hard_tanh", "hard_sigmoid", "power", "hswish", "gelu"}, default="sigmoid") – The activation function that is used in forget, reset, and input gates.
reverse_seq (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.
name (str, default=None) – The layer name.

Layer inputs:

the set of vector sequences, of dimensions:
- BatchLength - the length of a sequence
- BatchWidth * ListSize - the number of vector sequences in the input set
- Height * Width * Depth * Channels - the size of each vector in the sequence
(optional): the initial state of the LSTM before the first step. If this input is not specified, the initial state is all zeros. The dimensions:
- BatchLength should be 1
- the other dimensions should be the same as for the first input
(optional): the initial value of “previous output” to be used on the first step. If this input is not specified, all zeros will be used. The dimensions:
- BatchLength should be 1
- the other dimensions should be the same as for the first input

Layer outputs:

the layer output on every step
(optional): the cell state on every step

Both outputs have the following dimensions:

BatchLength, BatchWidth, ListSize are equal to the first input dimensions
Height, Width, Depth are equal to 1
Channels is equal to layer hidden size

property activation: Gets the activation function.

property dropout: Gets the dropout rate.

property hidden_size: Gets the hidden layer size.

property input_free_term: Gets the input free term.

property input_weights: Gets the input hidden layer weights. The blob size is (4*HiddenSize)x1x1xInputSize.

property recurrent_free_term: Gets the recurrent free term.

property recurrent_weights: Gets the recurrent hidden layer weights. The blob size is (4*HiddenSize)x1x1xHiddenSize.

property reverse_sequence: Checks if the input sequence will be reverted.

Qrnn

class neoml.Dnn.Qrnn(input_layers, pooling_type='f', hidden_size=1, window_size=1, stride=1, paddings=(0, 0), activation='tanh', dropout=0.0, mode='direct', name=None)

The quasi-recurrent layer that can be applied to a set of vector sequences. Unlike LSTM or GRU, the layer performs most of the calculations before the recurrent part, which helps improve performance on GPU. We use time convolution outside of the recurrent part instead of fully-connected layers in the recurrent part. See https://arxiv.org/abs/1611.01576

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
pooling_type (str, {"f", "fo", "ifo"}) – The pooling type used after recurrent
hidden_size (int, > 0) – The hidden layer size.
window_size (int, > 0) – The size of the window used in time convolution.
stride (int, > 0, default=1) – The window stride for time convolution.
paddings (tuple(int, int), >= 0, default=(0, 0)) – The additional zeros tacked to the start and end of sequence before convolution.
activation (str, {"linear", "elu", "relu", "leaky_relu", "abs", "sigmoid", "tanh", "hard_tanh", "hard_sigmoid", "power", "hswish", "gelu"}, default="tanh") – The activation function used in the update gate.
dropout (float, [0..1], default=0.0) – The dropout probability in the forget gate.
mode (str, {"direct", "reverse", "bidirectional_concat", "bidirectional_sum"}, default="direct") – The way of processing the input sequences. - bidirectional_concat means the direct and the reverse sequence are concatenated and then processed as one; - bidirectional_sum means the direct and the reverse sequence are added up and then processed as one.
name (str, default=None) – The layer name.

Layer inputs:

the set of vector sequences, of dimensions:
- BatchLength is the length of one sequence
- BatchWidth is the number of vector sequences in the set
- ListSize is 1
- Height * Width * Depth * Channels is the vector length
(optional): the initial state of the recurrent part. If not set, the recurrent part is all zeros before the first step. The dimensions:
- BatchLength, ListSize, Height, Width, Depth are 1
- BatchWidth is the same as for the first input
- Channels is hidden_size

Layer outputs:

the result sequence. The dimensions:
- BatchLength can be calculated from the input as (BatchLength + paddings[0] + paddings[1] - (window_size - 1))/(stride + 1)
- BatchWidth is the same as for the inputs
- ListSize, Height, Width, Depth are 1
- Channels is hidden_size for all recurrent modes except bidirectional_concat, when it is 2 * hidden_size

property activation: Gets the activation function used in the update gate.

property dropout: Gets the dropout probability for the forget gate.

property filter

Gets the trained weights for each gate. The blob dimensions:

BatchLength is 1
BatchWidth is 3 * hidden_size (contains the weights for each of the three gates in the order: update, forget, output)
Height is window_size
Width, Depth are 1
Channels is equal to the input’s Height * Width * Depth * Channels

property free_term: Gets the free term for all three gates in the same order. The blob size is 3 * hidden_size.

property hidden_size: Gets the hidden layer size.

property padding_back: Gets the size of zero padding at the sequence end.

property padding_front: Gets the size of zero padding at the sequence start.

property recurrent_mode: Gets the sequence processing mode.

property stride: Gets the stride for time convolution.

property window_size: Gets the window size for time convolution.

Gru

The gated recurrent unit.

class neoml.Dnn.Gru(input_layer, hidden_size, name=None)

The gated recurrent unit (GRU) layer that works with a set of vector sequences.

Parameters:

input_layer – The input layer and the number of its output. If no number is specified, the first output will be connected.
input_layer – object, tuple(object, int) or list of them
hidden_size (int) – The size of the hidden layer.
name (str, default=None) – The layer name.

Layer inputs:

the set of vector sequences. The dimensions:
- BatchLength is sequence length
- BatchWidth * ListSize is the number of sequences
- Height * Width * Depth * Channels is vector size
(optional): the initial previous step result. If you do not connect this input all zeros will be used on the first step. The dimensions are the same as for the first input.

Layer outputs:

a vector sequence of the same length. The dimensions:
- BatchLength, BatchWidth, ListSize equal to the input’s dimensions
- Height, Width, Depth are 1
- Channels is equal to hidden layer size

property gate_free_term: Gets the gate free terms as a blob of total 2 * hidden_size size.

property gate_weights

Gets the gate weights as a 2d matrix of the size:

BatchLength * BatchWidth * ListSize equal to 2 * hidden_size
Height * Width * Depth * Channels equal to this dimension of the input plus hidden_size

property hidden_size: Gets the hidden layer size.

property main_free_term: Gets the output free terms as a blob of total hidden_size size.

property main_weights

Gets the output weights as a 2d matrix of the size:

BatchLength * BatchWidth * ListSize equal to hidden_size
Height * Width * Depth * Channels equal to this dimension of the input plus hidden_size

Irnn

class neoml.Dnn.Irnn(input_layer, hidden_size=1, identity_scale=1.0, input_weight_std=0.001, reverse_seq=False, name=None)

IRNN implementation from this article: https://arxiv.org/pdf/1504.00941.pdf

It’s a simple recurrent unit with the following formula: \(Y_t = ReLU( FC_{input}( X_t ) + FC_{recur}( Y_t-1 ) )\) where \(FC\) are fully-connected layers.

The crucial point of this layer is weights initialization. The weight matrix of \(FC_{input}\) is initialized from N(0, input_weight_std), where input_weight_std is a layer parameter. The weight matrix of \(FC_{recur}\) is an identity matrix, multiplied by identity_scale parameter.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size.
identity_scale (float, default=1.) – The scale of identity matrix, used for the initialization of recurrent weights.
input_weight_std (float, default=1e-3) – The standard deviation for input weights.
reverse_seq (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.
name (str, default=None) – The layer name.

Layer inputs:

the set of vector sequences, of dimensions:
- BatchLength - the length of a sequence
- BatchWidth * ListSize - the number of vector sequences in the input set
- Height * Width * Depth * Channels - the size of each vector in the sequence

Layer outputs:

the result of the current step. The dimensions:
- BatchLength, BatchWidth, ListSize are the same as for the input
- Height, Width, Depth are 1
- Channels is hidden_size

property hidden_size: Gets the hidden layer size.

property identity_scale: Gets the multiplier for the identity matrix for initialization.

property input_free_term: Gets the FC_input free term, of hidden_size length.

property input_weight_std: Gets the standard deviation for input weights.

property input_weights

Gets the FC_input weights. The dimensions:

BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is the same as for the input

property recurrent_free_term: Gets the FC_recur free term, of hidden_size length.

property recurrent_weights

Gets the FC_recur weights. The dimensions:

BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is hidden_size

property reverse_sequence: Checks if the input sequence should be taken in reverse order.

IndRnn

class neoml.Dnn.IndRnn(input_layer, hidden_size=1, dropout_rate=0.0, reverse_sequence=False, activation='relu', name=None)

Independently Recurrent Neural Network (IndRNN): https://arxiv.org/pdf/1803.04831.pdf

It’s a simple recurrent unit with the following formula: \(Y_t = activation(W * X_t + B + U * dropout(Y_{t-1}))\) where \(W\) and \(B\) are weights and free terms of the fully-connected layer (\(W * X_t\) is a matrix multiplication) and \(U\) is a vector (\(U * Y_{t-1}\) is an eltwise multiplication of 2 vectors of the same length)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
hidden_size (int, default=1) – The size of hidden layer. Affects the output size.
dropout_rate (float, default=0.) – The rate of the dropout, applied to both input and recurrent data
activation (str, {'sigmoid', 'relu'}, default='relu') – The activation funciton applied to the output
reverse_sequence (bool, default=False) – Indicates if the input sequence should be taken in the reverse order.

Layer inputs:

the set of vector sequences, of dimensions:
- BatchLength - the sequence length
- BatchWidth * ListSize - the number of vector sequences in the input set
- Height * Width * Depth * Channels - the size of each vector in the sequence

Layer outputs:

the result of the layer. The dimensions:
- BatchLength, BatchWidth, ListSize are the same as for the input
- Height, Width, Depth are 1
- Channels is hidden_size

property activation: Gets the activation function.

property bias: Gets the bias vector, of hidden_size length.

property dropout_rate: Gets the dropout rate.

property hidden_size: Gets the hidden layer size.

property input_weights

Gets the input weights matrix. The dimensions:

BatchLength * BatchWidth * ListSize is hidden_size
Height * Width * Depth * Channels is the same as for the input

property recurrent_weights: Gets the recurrent weights vector, of hidden_size length.

property reverse_sequence: Checks if the input sequence should be taken in reverse order.

FullyConnected

class neoml.Dnn.FullyConnected(input_layers, element_count, is_zero_free_term=False, name=None)

The fully connected layer. It multiplies each of the input vectors by the weight matrix and adds the free term vector to the result.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
element_count (int, > 0) – The length of each vector in the output.
is_zero_free_term (bool, default=False) – If True, the free term vector is set to all zeros and not trained. If False, the free term is trained together with the weights.
name (str, default=None) – The layer name.

Layer inputs:

The layer can have any number of inputs. The dimensions:

BatchLength * BatchWidth * ListSize is the number of vectors
Height * Width * Depth * Channels is the vector size; should be the same for all inputs

Layer outputs:

The layer returns one output for each input. The dimensions:

BatchLength, BatchWidth, ListSize the same as for the input
Height, Width, Depth are 1
Channels is element_count

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property element_count: Gets the length of each vector in the output.

property free_term: Gets the free term vector, of element_count length.

property weights

Gets the trained weights as a blob of the dimensions:

BatchLength * BatchWidth * ListSize equal to element_count
Height, Width, Depth, Channels the same as for the first input

property zero_free_term: Sets the length of each vector in the output.

Activation layers

These layers calculate the value of different activation functions on their inputs.

Linear

class neoml.Dnn.Linear(input_layer, multiplier, free_term, name=None)

The layer that calculates a linear activation function for each element of a single input: \(f(x) = multiplier * x + free\_term\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
multiplier (float) – The linear function multiplier.
free_term (float) – The linear function free term.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property free_term: Gets the free term.

property multiplier: Gets the multiplier.

ELU

class neoml.Dnn.ELU(input_layer, alpha, name=None)

The layer that calculates the ELU activation function for each element of the single input:

\(f(x) = alpha * (e^x - 1)\) if \(x < 0\)
\(f(x) = x\) if \(x \ge 0\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
alpha (float) – The multiplier before the exponential function used for negative values of x.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property alpha: Gets the exponent multiplier.

ReLU

class neoml.Dnn.ReLU(input_layer, threshold=0.0, name=None)

The layer that calculates the ReLU activation function for each element of the single input:

\(f(x) = 0\) if \(x \le 0\)
\(f(x) = x\) if \(x > 0\)

You also can set the cutoff upper threshold:

\(f(x) = 0\) if \(x \le 0\)
\(f(x) = x\) if \(0 < x < threshold\)
\(f(x) = threshold\) if \(threshold \le x\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
threshold (float, default=0) – The upper cutoff threshold. 0 resets the threshold.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property threshold: Gets the upper cutoff threshold.

LeakyReLU

class neoml.Dnn.LeakyReLU(input_layer, alpha, name=None)

The layer that calculates the “leaky” ReLU activation function for each element of the single input:

\(f(x) = alpha * x\) if \(x \le 0\)
\(f(x) = x\) if \(x > 0\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
alpha (float, default=0) – The multiplier used for negative values of x.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property alpha: Gets the multiplier used for negative values of x.

HSwish

class neoml.Dnn.HSwish(input_layer, name=None)

The layer that calculates the H-Swish activation function for each element of the single input:

\(f(x) = 0\) if \(x \le -3\)
\(f(x) = x * (x + 3) / 6\) if \(-3 < x < 3\)
\(f(x) = x\) if \(x \ge 3\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

GELU

class neoml.Dnn.GELU(input_layer: Layer | Tuple[Layer, int] | neoml.PythonWrapper.GELU, calculation_mode: Literal['precise', 'sigmoid_approximate'] = 'sigmoid_approximate', name: str = None)

The layer that calculates the GELU activation function for each element of the single input: \(f(x) = x * 0.5 * ( 1 + erf( x / sqrt(2) ) )\) or approx.: f(x) = x * sigmoid( 1.702 * x )

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property calculation_mode: Literal['precise', 'sigmoid_approximate']: ‘precise’ (calculate GELU using the error function) or ‘sigmoid_approximate’ (using an approximation x * sigmoid(1.702x))

Abs

class neoml.Dnn.Abs(input_layer, name=None)

The layer that calculates the absolute value of each element of the single input.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

Sigmoid

class neoml.Dnn.Sigmoid(input_layer, name=None)

The layer that calculates the sigmoid activation function for each element of the signle input: \(f(x) = 1 / (1 + e^{-x})\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

Tanh

class neoml.Dnn.Tanh(input_layer, name=None)

The layer that calculates the tanh activation function for each element of the single input: \(f(x) = (e^{2 * x} - 1) / (e^{2 * x} + 1)\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

HardTanh

class neoml.Dnn.HardTanh(input_layer, name=None)

The layer that calculates the HardTanh activation function for each element of the single input:

\(f(x) = -1\) if \(x \le -1\)
\(f(x) = x\) if \(-1 < x < 1\)
\(f(x) = 1\) if \(x \ge 1\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

HardSigmoid

class neoml.Dnn.HardSigmoid(input_layer, slope, bias, name=None)

The layer that calculates the “hard sigmoid” activation function for each element of the single input:

\(f(x) = 0\) if \(x \le -bias / slope\)
\(f(x) = slope * x + bias\) if \(-bias / slope < x < (1 - bias) / slope\)
\(f(x) = 1\) if \(x \ge (1 - bias) / slope\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
slope (float, > 0) – The slope of the linear component.
bias (float) – The shift of the linear component from 0.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property bias: Gets the shift of the linear component from 0.

property slope: Gets the slope of the linear component.

Power

class neoml.Dnn.Power(input_layer, exponent, name=None)

The layer that raises each element of the input to the given power.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
exponent (float) – The power to which the input elements will be raised.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with activation function values on each of the input elements

property exponent: Gets the power to which the input elements will be raised.

Exp

class neoml.Dnn.Exp(input_layer, name=None)

The layer that calculates the exponent function for each element of the single input.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with exponent function values on each of the input elements

Log

class neoml.Dnn.Log(input_layer, name=None)

The layer that calculates the logarithm function for each element of the single input.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with logarithm function values on each of the input elements

Erf

class neoml.Dnn.Erf(input_layer, name=None)

The layer that calculates the error function for each element of the single input.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

a data blob of the same size as the input, with error function values on each of the input elements

Convolutional layers

The layers that perform various types of convolution.

Conv

class neoml.Dnn.Conv(input_layers, filter_count=1, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), dilation_size=(1, 1), is_zero_free_term=True, name=None)

The layer that performs convolution on a set of two-dimensional multi-channel images.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size (tuple(int, int), default=(3, 3)) – Filter size (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size (tuple(int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
dilation_size (tuple(int, int), default=(1, 1)) – The step values for dilated convolution (vertical, horizontal). The default value of (1, 1) means no dilation.
is_zero_free_term (bool, default=True) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength * BatchWidth * ListSize - the number of images in the set
Height - the images’ height
Width - the images’ width
Depth * Channels - the number of channels the image format uses

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height - (1 + DilationHeight * (FilterHeight - 1))) / StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width - (1 + DilationWidth * (FilterWidth - 1))) / StrideWidth + 1
Depth is equal to 1
Channels is equal to the number of filters

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property dilation_size: Gets the step values for dilated convolution.

property filter

Gets the filters. The dimensions:

BatchLength * BatchWidth * ListSize is filter_count
Height, Width are taken from filter_size
Depth, Channels are equal to the inputs’ dimensions

property filter_count: Gets the number of filters.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is filter_count.

property padding_size: Gets the padding size.

property stride_size: Gets the convolution stride.

Conv3D

class neoml.Dnn.Conv3D(input_layers, filter_count=1, filter_size=(3, 3, 3), stride_size=(1, 1, 1), padding_size=(0, 0, 0), is_zero_free_term=False, name=None)

The layer that performs convolution on a set of three-dimensional multi-channel images.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int, int), default=(3, 3, 3)) – Filter size (height, width, depth).
stride_size ((int, int, int), default=(1, 1, 1)) – Convolution stride (vertical, horizontal, depth).
padding_size ((int, int, int), default=(0, 0, 0)) – The size of the padding (vertical, horizontal, depth).
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength * BatchWidth * ListSize - the number of images in the set
Height - the images’ height
Width - the images’ width
Depth - the images’ depth
Channels - the number of channels the image format uses

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height - FilterHeight) / StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width - FilterWidth) / StrideWidth + 1
Depth can be calculated from the input Depth as (2 * PaddingDepth + Depth - FilterDepth) / StrideDepth + 1
Channels is equal to the number filters

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property filter

Gets the filters. The dimensions:

BatchLength * BatchWidth * ListSize is filter_count
Height, Width, Depth are taken from filter_size
Channels is equal to the inputs’ Channels

property filter_count: Gets the number of filters.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is filter_count.

property padding_size: Gets the padding size.

property stride_size: Gets the convolution stride.

TransposedConv3D

class neoml.Dnn.TransposedConv3D(input_layers, filter_count=1, filter_size=(3, 3, 3), stride_size=(1, 1, 1), padding_size=(0, 0, 0), is_zero_free_term=False, name=None)

The layer that performs transposed convolution on a set of three-dimensional multi-channel images.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int, int), default=(3, 3, 3)) – Filter size (height, width, depth).
stride_size ((int, int, int), default=(1, 1, 1)) – Convolution stride (vertical, horizontal, depth).
padding_size ((int, int, int), default=(0, 0, 0)) – The size of the padding (vertical, horizontal, depth).
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength * BatchWidth * ListSize - the number of images in the set
Height - the images’ height
Width - the images’ width
Depth - the images’ depth
Channels - the number of channels the image format uses

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as StrideHeight * (Height - 1) + FilterHeight - 2 * PaddingHeight
Width can be calculated from the input Width as StrideWidth * (Width - 1) + FilterWidth - 2 * PaddingWidth
Depth can be calculated from the input Depth as StrideDepth * (Depth - 1) + FilterDepth - 2 * PaddingDepth
Channels is equal to the number of filters

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property filter

Gets the filters. The dimensions:

BatchLength, ListSize are 1
BatchWidth is equal to the inputs’ Channels
Height, Width, Depth are taken from filter_size
Channels is filter_count

property filter_count: Gets the number of filters.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is filter_count.

property padding_size: Gets the padding size.

property stride_size: Gets the convolution stride.

TransposedConv

class neoml.Dnn.TransposedConv(input_layers, filter_count=1, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), dilation_size=(1, 1), is_zero_free_term=False, name=None)

The layer that performs transposed convolution on a set of two-dimensional multi-channel images.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of filters.
filter_size ((int, int), default=(3, 3)) – Filter size (height, width).
stride_size ((int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size ((int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
dilation_size ((int, int), default=(1, 1)) – The step values for dilated convolution (vertical, horizontal). The default value of (1, 1) means no dilation.
is_zero_free_term (bool, default=False) – Specifies if the free term should be zero.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength * BatchWidth * ListSize - the number of images in the set
Height - the images’ height
Width - the images’ width
Depth * Channels - the number of channels the image format uses

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as StrideHeight * (Height - 1) + (FilterHeight - 1) * DilationHeight + 1 - 2 * PaddingHeight
Width can be calculated from the input Width as StrideWidth * (Width - 1) + (FilterWidth - 1) * DilationWidth + 1 - 2 * PaddingWidths
Depth is 1
Channels is equal to the number of filters

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property dilation_size: Gets the step values for dilated convolution.

property filter

Gets the filters. The dimensions:

BatchLength, ListSize are 1
BatchWidth is equal to the inputs’ Channels * Depth
Height, Width are taken from filter_size
Channels is filter_count

property filter_count: Gets the number of filters.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is filter_count.

property padding_size: Gets the padding size.

property stride_size: Gets the convolution stride.

ChannelwiseConv

class neoml.Dnn.ChannelwiseConv(input_layers, filter_count, filter_size=(3, 3), stride_size=(1, 1), padding_size=(0, 0), is_zero_free_term=False, name=None)

The layer that performs channel-wise convolution. Each channel of the input is convolved with the corresponding channel of the filter.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, default=1) – The number of channels in the filter. Should be the same as the number of channels in the input.
filter_size ((int, int), default=(3, 3)) – Filter size (height, width).
stride_size ((int, int), default=(1, 1)) – Convolution stride (vertical, horizontal).
padding_size ((int, int), default=(0, 0)) – The size of the padding (vertical, horizontal).
free_term (bool, default=True) – Specifies if the free term should be non-zero.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength * BatchWidth * ListSize - the number of images in the set
Height - the images’ height
Width - the images’ width
Depth * Channels - the number of channels the image format uses

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength, BatchWidth, ListSize are equal to the input dimensions
Height can be calculated from the input Height as (2 * PaddingHeight + Height - FilterHeight)/StrideHeight + 1
Width can be calculated from the input Width as (2 * PaddingWidth + Width - FilterWidth)/StrideWidth + 1
Depth is equal to 1
Channels is equal to the number of channels in the filter and the input

apply_batch_normalization(layer)

Applies batch normalization to this layer. Batch normalization must be deleted from the dnn afterwards and layers which were connected to the batch norm must be connected to this layer.

Parameters:: layer (neoml.Dnn.BatchNormalization) – batch norm to be applied

property filter

Gets the filter. The dimensions:

BatchLength, BatchWidth, ListSize, Depth are 1
Height, Width are taken from filter_size
Channels is equal to the inputs’ Channels

property filter_count: Gets the number of channels in the filter.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is inputs’ Channels.

property padding_size: Gets the padding size.

property stride_size: Gets the convolution stride.

TimeConv

class neoml.Dnn.TimeConv(input_layers, filter_count, filter_size, padding_front=0, padding_back=0, dilation=1, stride=1, name=None)

The layer that performs time convolution on a set of sequences.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
filter_count (int, > 0) – The number of filters.
filter_size (int, > 0) – Filter size.
padding_front (int, >= 0) – The number of zeros to be added at the sequence start.
padding_back (int, >= 0) – The number of zeros to be added at the sequence end.
dilation (int, default=1) – The step value for dilated convolution. 1 means no dilation.
stride (int, > 0, default=1) – Convolution stride.
name (str, default=None) – The layer name.

Layer inputs:

Can have several inputs, of the dimensions:

BatchLength is the sequence length
BatchWidth * ListSize - the number of sequences in the set
Height * Width * Depth * Channels - the size of each element

Layer outputs:

The layer has as many outputs as the inputs, of the dimensions:

BatchLength can be calculated from the input BatchLength as (padding_front + padding_back + BatchLength - (1 + dilation * (filter_size - 1)))/stride + 1
BatchWidth, ListSize are equal to the input dimensions
Height, Width, Depth are equal to 1
Channels is equal to the number of filters

property dilation: Gets the step value for dilated convolution.

property filter

Gets the filters. The dimensions:

BatchLength is 1
BatchWidth is filter_count
Height is filter_size
Width, Depth are 1
Channels is the inputs’ Height * Width * Depth * Channels

property filter_count: Gets the number of filters.

property filter_size: Gets the filter size.

property free_term: Gets the free term. The blob size is filter_count.

property padding_back: Gets the padding size.

property padding_front: Gets the padding size.

property stride: Gets the convolution stride.

Loss layers

These layers calculate different types of loss functions on the network response. They have no output and return the loss value through a last_loss method.

CrossEntropyLoss

class neoml.Dnn.CrossEntropyLoss(input_layers, softmax=True, loss_weight=1.0, name=None)

The layer that calculates the loss value as cross-entropy between the result and the standard: \(loss = -\sum{y_i * \log{z_i}}\), where for each i class \(y_i\) represents the class label, \(z_i\) is the network response (with softmax applied or not)

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
softmax (bool, default=True) – Specifies if softmax function should be applied to the result.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of objects
- Height * Width * Depth * Channels - the number of classes
the correct class labels. Two formats are acceptable:
- The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
- The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

property apply_softmax: Checks if softmax function should be applied to the result.

BinaryCrossEntropyLoss

class neoml.Dnn.BinaryCrossEntropyLoss(input_layers, positive_weight=1.0, loss_weight=1.0, name=None)

The layer that calculates the cross-entropy loss function for binary classification: \(loss = - y * \log(sigmoid(x)) - (1 - y) * \log(1 - sigmoid(x))\), where x is the network response, y is the correct class label (can be -1 or 1)

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
positive_weight (float, default=1.0) – The multiplier for correctly classified objects. Tune this value to prioritize precision (positive_weight < 1) or recall (positive_weight > 1) during training.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or -1).
(optional): the objects’ weights.

The dimensions of all inputs are the same:

BatchLength * BatchWidth * ListSize - the number of objects

Height, Width, Depth, Channels should be equal to 1

Layer outputs:

The layer has no output.

property positive_weight: Gets the multiplier for the term that corresponds to the correct results.

EuclideanLoss

class neoml.Dnn.EuclideanLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates the loss function equal to Euclidean distance between the network response and the correct classes.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss.
the correct class objects.
(optional): the objects’ weights.

The dimensions of all inputs are the same:

BatchLength * BatchWidth * ListSize - the number of objects

Height * Width * Depth * Channels - the object size

Layer outputs:

The layer has no output.

HingeLoss

class neoml.Dnn.HingeLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates hinge loss function for binary classification: \(f(x) = \max(0, 1 - x * y)\), where x is the network response, y is the correct class label (1 or -1).

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or -1).
(optional): the objects’ weights.

The dimensions of all inputs are the same:

BatchLength * BatchWidth * ListSize - the number of objects

Height, Width, Depth, Channels should be equal to 1

Layer outputs:

The layer has no output.

SquaredHingeLoss

class neoml.Dnn.SquaredHingeLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates squared hinge loss function for binary classification:

\(f(x) = -4 * x * y\) if \(x * y < -1\)
\(f(x) = (\max(0, 1 - x * y))^2\) if \(x * y \ge -1\)

where: x is the network response, y is the correct class label (1 or -1).

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or -1).
(optional): the objects’ weights.

The dimensions of all inputs are the same:

BatchLength * BatchWidth * ListSize - the number of objects

Height, Width, Depth, Channels should be equal to 1

Layer outputs:

The layer has no output.

FocalLoss

class neoml.Dnn.FocalLoss(input_layers, force, loss_weight=1.0, name=None)

The layer that calculates the focal loss function for multiple class classification. It is a version of cross-entropy in which the easily-distinguished objects receive smaller penalties. This helps focus on learning the difference between similar-looking elements of different classes.

\(f(x) = -(1 - x_{right})^{force} * \log(x_{right})\) where \(x_{right}\) is the network response element that represents the probability for the object to belong to the correct class.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
force (float, > 0, default=2.0) – The focal force, that is, the degree to which learning will concentrate on similar objects.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:

BatchLength * BatchWidth * ListSize - the number of objects
Height * Width * Depth * Channels - the number of classes

the correct class labels. Two formats are acceptable:
- The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
- The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

property force: Gets the focal force multiplier.

BinaryFocalLoss

class neoml.Dnn.BinaryFocalLoss(input_layers, force, loss_weight=1.0, name=None)

The layer that calculates the focal loss function for binary classification. It is a version of cross-entropy in which the easily-distinguished objects receive smaller penalties. This helps focus on learning the difference between similar-looking elements of different classes.

\(f(x) = -(sigmoid(-y * x))^{force} * \log(1 + e^{-y * x})\) where: x is the network response, y is the correct class label (1 or -1).

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
force (float, > 0, default=2.0) – The focal force, that is, the degree to which learning will concentrate on similar objects.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes.
the correct class labels (1 or -1).
(optional): the objects’ weights.

The dimensions of all inputs are the same:

BatchLength * BatchWidth * ListSize - the number of objects

Height, Width, Depth, Channels should be equal to 1

Layer outputs:

The layer has no output.

property force: Gets the focal force multiplier.

CenterLoss

class neoml.Dnn.CenterLoss(input_layers, class_count, rate, loss_weight=1.0, name=None)

The layer that penalizes large differences between objects of the same class. See the paper at http://ydwen.github.io/papers/WenECCV16.pdf

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
class_count (int) – The number of classes in the model.
rate (float, [0..1]) – Class center convergence rate: the multiplier used for calculating the moving mean of the class centers for each subsequent iteration.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of objects
- Height * Width * Depth * Channels - the number of classes
the correct class labels. Two formats are acceptable:
- The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
- The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

property class_count: Gets the number of classes in the model.

property rate: Gets the convergence rate multiplier.

MultiHingeLoss

class neoml.Dnn.MultiHingeLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates hinge loss function for multiple class classification: \(f(x) = \max(0, 1 - (x_{right} - x_{max\_wrong}))\) where \(x_{right}\) is the network response for the correct class, \(x_{max\_wrong}\) is the largest response for all the incorrect classes.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of objects
- Height * Width * Depth * Channels - the number of classes
the correct class labels. Two formats are acceptable:
- The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
- The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

MultiSquaredHingeLoss

class neoml.Dnn.MultiSquaredHingeLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates squared hinge loss function for multiple class classification:

\(f(x) = -4 * (x_{right} - x_{max\_wrong})\) if \(x_{right} - x_{max\_wrong} < -1\)
\(f(x) = (\max(0, 1 - (x_{right} - x_{max\_wrong})))^2\) if \(x_{right} - x_{max\_wrong} \ge -1\)

where \(x_{right}\) is the network response for the correct class, \(x_{max\_wrong}\) is the largest response for all the incorrect classes.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of objects
- Height * Width * Depth * Channels - the number of classes
the correct class labels. Two formats are acceptable:
- The blob contains float data, the dimensions are equal to the first input dimensions. It should be filled with zeros, and only the coordinate of the class to which the corresponding object from the first input belongs should be 1.
- The blob contains int data with BatchLength, BatchWidth, and ListSize equal to these dimensions of the first input, and the other dimensions equal to 1. Each object in the blob contains the number of the class to which the corresponding object from the first input belongs.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

CustomLoss

NeoML provides an interface for user-implemented custom loss functions. They must be constructed out of simple arithmetic and Autodifferentiation functions.

class neoml.Dnn.CustomLossCalculatorBase

The base class that you should implement to calculate the custom loss function.

abstract calc(data, labels)

Calculates the custom loss function. This function may use only the operations supported for autodiff:

simple arithmetic: + - * /
the neoml.AutoDiff.* functions
neoml.Autodiff.const for creating additional blobs filled with given values

Parameters:: data (neoml.Blob.Blob) – the network response with the probability

distribution of objects over classes. The blob dimensions:

BatchLength - the number of objects
Channels - the object size
all other dimensions equal to 1

Parameters:: labels (neoml.Blob.Blob) – the correct labels, of the same dimensions

as the first blob, and containing 1 in the coordinate of the class to which the corresponding object belongs, 0 in all other places.

Returns:: a blob that contains the loss function values

for each object in the batch. This blob will have the same BatchLength as the input blobs, and all its other dimensions should be 1. :rtype: neoml.Blob.Blob

class neoml.Dnn.CustomLoss(input_layers, loss_calculator=None, loss_weight=1.0, name=None)

The layer that calculates a custom loss function.

Parameters:

input_layers (list of object, tuple(object, int)) – the input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_calculator (neoml.Dnn.CustomLossCalculatorBase) – a user-implemented object that provides the method to calculate the custom loss.
loss_weight (float, default=1.0) – the multiplier for the loss function value during training.
name (str, default=None) – the layer name.

Layer inputs:

the network response for which you are calculating the loss. It should contain the probability distribution for objects over classes. If you are not going to apply softmax in this layer, each element should already be >= 0, and the sum over Height * Width * Depth * Channels dimension should be equal to 1. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of objects
- Height * Width * Depth * Channels - the number of classes
the correct class labels. The blob of the same dimensions as the first input, filled with zeros, where only the coordinate of the class to which the corresponding object from the first input belongs is be 1.
(optional): the objects’ weights. The dimensions:
- BatchLength, BatchWidth, ListSize should be the same as for the first input
- the other dimensions should be 1

Layer outputs:

The layer has no output.

Pooling layers

The layers that perform pooling operations on the input data.

MaxPooling

class neoml.Dnn.MaxPooling(input_layers, filter_size=(3, 3), stride_size=(1, 1), name=None)

The pooling layer that finds maximum in a window.

Parameters:

input_layers ((object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int), default=(3, 3)) – The size of the window: (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Window stride (vertical, horizontal).
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth * Channels - the number of channels the image format uses

Layer outputs:

the result of pooling The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height can be calculated from the input as (Height - FilterHeight)/StrideHeight + 1
- Width can be calculated from the input as (Width - FilterWidth)/StrideWidth + 1
- Depth and Channels are equal to the input dimensions

MeanPooling

class neoml.Dnn.MeanPooling(input_layers, filter_size=(3, 3), stride_size=(1, 1), name=None)

The pooling layer that takes average over the window.

Parameters:

input_layers ((object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int), default=(3, 3)) – The size of the window: (height, width).
stride_size (tuple(int, int), default=(1, 1)) – Window stride (vertical, horizontal).
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth * Channels - the number of channels the image format uses

Layer outputs:

the result of pooling The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height can be calculated from the input as (Height - FilterHeight)/StrideHeight + 1
- Width can be calculated from the input as (Width - FilterWidth)/StrideWidth + 1
- Depth and Channels are equal to the input dimensions

GlobalMaxPooling

class neoml.Dnn.GlobalMaxPooling(input_layers, max_count, name=None)

The layer that finds maximum over the whole three-dimensional image, allowing for multiple largest elements to be found. If you set the number of largest elements to 1, it will function exactly as MaxPooling3d with the filter size equal to the input image size.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
max_count (int, > 0) – The number of largest elements to be found. Note that these do not have to be equal to each other; the top max_count elements will be returned.
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth - the images’ depth
- Channels - the number of channels the image format uses

Layer outputs:

the maximum values found. The dimensions:
- BatchLength, BatchWidth, ListSize, Channels are equal to the input dimensions
- Height, Depth are 1
- Width is max_count
(optional): the indices of the values found in the input blob. The dimensions are the same.

property max_count: Gets the number of largest elements to be found.

GlobalMeanPooling

class neoml.Dnn.GlobalMeanPooling(input_layers, name=None)

The layer that finds the average over the whole three-dimensional image. It functions exactly as MeanPooling3d with the filter size equal to the input image size.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth - the images’ depth
- Channels - the number of channels the image format uses

Layer outputs:

the average values over each image. The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height, Width, Depth are 1
- Channels is equal to the input Channels

GlobalSumPooling

class neoml.Dnn.GlobalSumPooling(input_layers, name=None)

The layer that finds the sum over the whole three-dimensional image.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth - the images’ depth
- Channels - the number of channels the image format uses

Layer outputs:

the sums over each image. The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height, Width, Depth are 1
- Channels is equal to the input Channels

MaxOverTimePooling

class neoml.Dnn.MaxOverTimePooling(input_layers, filter_len, stride_len, name=None)

The layer that finds maximums on the set of sequences, with the window taken over BatchLength axis.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_len (int) – The window size. Set to <= 0 if the maximum over the whole sequence length should be found.
stride_len (int, > 0) – The window stride. Meaningful only for filter_len > 0.
name (str, default=None) – The layer name.

Layer inputs:

the set of sequences, of dimensions:
- BatchLength is the sequence length
- BatchWidth * ListSize is the number of sequences in the set
- Height * Width * Depth * Channels is the length of each vector

Layer outputs:

the pooling result. The dimensions:
- BatchLength is:
  - 1 if filter_len is <= 0
  - (BatchLength - filter_len) / stride_len + 1 otherwise
- the other dimensions are the same as for the input

property filter_len: Gets the window size.

property stride_len: Gets the window stride.

ProjectionPooling

class neoml.Dnn.ProjectionPooling(input_layers, dimension, original_size, name=None)

The layer that takes the average over one of the dimensions. This dimension is either compressed to a point when original_size=False, or stays the same length with all elements equal to the average when original_size=True.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – The dimension along which the average is taken.
original_size (bool, default=False) – Specifies if the blob should stay the same size, with the average value broadcast along the pooling dimension.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size.

Layer outputs:

the result of pooling. The dimensions:
- all stay the same if original_size is True
- if original_size is False, the pooling dimension is 1 and other dimensions stay the same

property dimension: Gets the dimension along which the average is to be calculated.

property original_size: Checks if the blob will stay the same size, with the average value broadcast along the pooling dimension.

MaxPooling3D

class neoml.Dnn.MaxPooling3D(input_layers, filter_size=(3, 3, 3), stride_size=(1, 1, 1), name=None)

The pooling layer that finds maximum in a window for three-dimensional images.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int, int), default=(3, 3, 3)) – The size of the window: (height, width, depth).
stride_size (tuple(int, int, int), default=(1, 1, 1)) – Window stride (vertical, horizontal, depth).
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth - the images’ depth
- Channels - the number of channels the image format uses

Layer outputs:

the result of pooling The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height can be calculated from the input as (Height - FilterHeight)/StrideHeight + 1
- Width can be calculated from the input as (Width - FilterWidth)/StrideWidth + 1
- Depth can be calculated from the input as (Depth - FilterDepth)/StrideDepth + 1
- Channels is equal to the input Channels

MeanPooling3D

class neoml.Dnn.MeanPooling3D(input_layers, filter_size=(3, 3, 3), stride_size=(1, 1, 1), name=None)

The pooling layer that takes average over a window for three-dimensional images.

Parameters:

input_layers (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
filter_size (tuple(int, int, int), default=(3, 3, 3)) – The size of the window: (height, width, depth).
stride_size (tuple(int, int, int), default=(1, 1, 1)) – Window stride (vertical, horizontal, depth).
name (str, default=None) – The layer name.

Layer inputs:

the set of images, of dimensions:
- BatchLength * BatchWidth * ListSize - the number of images in the set
- Height - the images’ height
- Width - the images’ width
- Depth - the images’ depth
- Channels - the number of channels the image format uses

Layer outputs:

the result of pooling The dimensions:
- BatchLength, BatchWidth, ListSize are equal to the input dimensions
- Height can be calculated from the input as (Height - FilterHeight)/StrideHeight + 1
- Width can be calculated from the input as (Width - FilterWidth)/StrideWidth + 1
- Depth can be calculated from the input as (Depth - FilterDepth)/StrideDepth + 1
- Channels is equal to the input Channels

Softmax

class neoml.Dnn.Softmax(input_layer, area='object_size', name=None)

The layer that calculates softmax function on each vector of a set: softmax(x[0], … , x[n-1])[i] = exp(x[i]) / (exp(x[0]) + … + exp(x[n-1]))

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
area (str, {"object_size", "batch_length", "list_size", "channel"}, default="object_size") –
Specifies which dimensions constitute the vector length:
- object_size: there are BatchLength * BatchWidth * ListSize vectors, of Height * Width * Depth * Channels length each
- batch_length: there are BatchWidth * ListSize * Height * Width * Depth * Channels vectors, of BatchLength length each
- list_size: there are BatchLength * BatchWidth * Height * Width * Depth * Channels vectors, of ListSize length each
- channel: there are BatchLength * BatchWidth * ListSize * Height * Width * Depth vectors, of Channels length each
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size. The area setting determines which dimensions would be considered to constitute vector length.

Layer outputs:

a blob of the same size with softmax applied to every vector.

property area: Checks which dimensions constitute the vector length.

Dropout

class neoml.Dnn.Dropout(input_layer, rate=0.5, spatial=False, batchwise=False, name=None)

The layer that randomly sets some elements of the single input to 0. If the input BatchLength is greater than 1, all elements along the same BatchLength coordinate will use the same mask. When the network is not being trained (for example, during a test run), the dropout will not happen.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
rate (float, [0..1]) – The proportion of elements that will be set to 0.
spatial (bool, default=False) – Turns on and off the spatial dropout mode. When True, the whole contents of a channel will be filled with zeros, instead of elements one by one. Useful for convolutional networks.
batchwise (bool, default=False) – Turns on and off the batchwise dropout mode. When True, the same mask will be used along the same BatchWidth. Useful for large input size.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any dimensions.

Layer outputs:

a blob of the same dimensions, with some of the elements set to 0, during training only. When you run the network, this layer does nothing.

property batchwise: Checks if the batchwise mode is on.

property rate: Gets the dropout rate.

property spatial: Checks if the spatial mode is on.

CumSum

class neoml.Dnn.CumSum(input_layer, dimension='channels', reverse=False, name=None)

The layer that calculates cumulative sum along the given dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {'batch_length', 'batch_width', 'list_size', 'height', 'width', 'depth', 'channels'}, default='channels') – The dimension along which the cumulative sum is to be calculated.
reverse (bool, default=False) – If True then cumulative sums will be calculated in reverse order
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size and data type

Layer outputs:

a blob of the same size and data type with the cumulative sums

property dimension: Gets the dimension along which the cumulative sum is to be calculated.

property reverse: Gets the flag of reversed sum order

Normalization layers

BatchNormalization

class neoml.Dnn.BatchNormalization(input_layer, channel_based=True, zero_free_term=False, slow_convergence_rate=1.0, name=None)

The layer that performs normalization using the formula: \(bn(x)[i][j] = ((x[i][j] - mean[j]) / \sqrt{var[j]}) * gamma[j] + beta[j]\)

gamma and beta are the trainable parameters
mean and var depend on whether the layer is being trained:
- If the layer is being trained, mean[j] and var[j] are the mean value and the variance of x data with j coordinate across all i.
- If the layer is not being trained, mean[j] and var[j] are the exponential moving mean and the unbiased variance estimate calculated during training.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
channel_based (bool, default=True) –
Turns on and off channel-based statistics:
- If True, mean, var, gamma, and beta in the formula will be vectors of the input Channels length. The i coordinate will iterate over all values from 0 to BatchLength * BatchWidth * ListSize * Height * Width * Depth - 1.
- If False, the mean, var, gamma, and beta vectors will have the Height * Width * Depth * Channels length. The i coordinate will iterate over all values from 0 to BatchLength * BatchWidth * ListSize - 1.
zero_free_term (bool, default=False) – Specifies if the free term (beta) should be trained or filled with zeros.
slow_convergence_rate (float, default=1.0) – The coefficient for calculating the exponential moving mean and variance.
name (str, default=None) – The layer name.

property channel_based: Gets the channel-based mode.

property final_params

Gets the trained parameters as a blob of the dimensions:

BatchLength, ListSize, Channels equal to 1
BatchWidth is 2
Height, Width, Depth equal to 1 if in channel-based mode, otherwise the same as the dimensions of the input

property slow_convergence_rate: Gets the coefficient for calculating the exponential moving mean and variance.

property use_final_params_for_init: Checks if the final parameters should be used for initialization.

property zero_free_term: Indicates if the free term should be zero.

ObjectNormalization

class neoml.Dnn.ObjectNormalization(input_layer, epsilon=1e-05, name=None)

The layer that performs object normalization using the formula: \(objectNorm(x)[i][j] = ((x[i][j] - mean[i]) / \sqrt{var[i] + epsilon}) * scale[j] + bias[j]\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
epsilon (float, default=0.00001) – The small value added to the variance to avoid division by zero.
name (str, default=None) – The layer name.

Layer inputs:

a set of objects. The dimensions:
- BatchLength * BatchWidth * ListSize is the number of objects
- Height * Width * Depth * Channels is the object size

Layer outputs:

the normalized result, of the same size as the input.

property bias: Gets bias, one of the trainable parameters in the formula. The total blob size is equal to the input Height * Width * Depth * Channels.

property epsilon: Gets the small value used to avoid division by zero.

property scale: Gets scale, one of the trainable parameters in the formula. The total blob size is equal to the input Height * Width * Depth * Channels.

Lrn (Local Response Normalization)

class neoml.Dnn.Lrn(input_layer, window_size=1, bias=1.0, alpha=0.0001, beta=0.75, name=None)

Lrn layer performs local response normlization with the following formula: \(LRN(x)[obj][ch] = x[obj][ch] * / ((bias + alpha * sqrSum[obj][ch] / windowSize) ^ beta)\) where \(obj\) is index of the object , \(ch\) is index of the channel, \(window_size\), \(bias\), \(alpha\) and \(beta\) are layer settings and \(sqrSum[obj][ch] = \sum_{i=\max(0, ch - \lfloor\frac{windowSize - 1}{2}\rfloor)}^{\min(C - 1, ch + \lceil\frac{windowSize - 1}{2}\rceil)}x[obj][i]^2\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
window_size – The size of window used in normalization
bias – value added to the scaled sum of squares
alpha – scale value of sum of squares
beta – exponent of the formula

Type:

int, default=1

Type:

float, default=1.

Type:

float, default=1e-4

Type:

float, default=0.75

Layer inputs:

the set of objects, of dimensions:
- BatchLength * BatchWidth * ListSize * Height * Width * Depth - the number of objects
- Channels - the size of the object

Layer outputs:

the result of the layer, of the dimensions of the input.

property alpha: Gets the alpha.

property beta: Gets the beta.

property bias: Gets the bias.

property window_size: Gets the window size.

Elementwise operation layers

EltwiseSum

class neoml.Dnn.EltwiseSum(input_layers, name=None)

The layer that adds up its inputs element by element.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have at least two inputs. All inputs should be the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

EltwiseSub

class neoml.Dnn.EltwiseSub(input_layers, name=None)

The layer that substracts the second input from the first element by element.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have at least two inputs. All inputs should be the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

EltwiseMul

class neoml.Dnn.EltwiseMul(input_layers, name=None)

The layer that multiplies its inputs element by element.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have at least two inputs. All inputs should be the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

EltwiseDiv

class neoml.Dnn.EltwiseDiv(input_layers, name=None)

The layer that divides its inputs element by element.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have two inputs of the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

EltwiseNegMul

class neoml.Dnn.EltwiseNegMul(input_layers, name=None)

The layer that that calculates the element-wise product of 1 - x, where x is the element of the first input, and the corresponding elements of all other inputs.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have at least two inputs. All inputs should be the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

EltwiseMax

class neoml.Dnn.EltwiseMax(input_layers, name=None)

The layer that finds the maximum among the elements that are at the same position in all input blobs.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer should have at least two inputs. All inputs should be the same size.

Layer outputs:

The layer has one output of the same size as the inputs.

Quality control layers

Accuracy

class neoml.Dnn.Accuracy(input_layers, reset=True, name=None)

The layer that calculates classification accuracy, that is, the proportion of objects classified correctly in the set.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.

Layer inputs:

a blob with the network response The dimensions:
- BatchLength * BatchWidth * ListSize equal to the number of objects that were classified
- Height, Width, Depth equal to 1
- Channels equal to 1 for binary classification and to the number of classes if there are more than 2
a blob with the correct class labels The dimensions should be the same as for the first input

Layer outputs:

a blob with only one element, which contains the proportion of correctly classified objects among all objects

property reset: Checks if the statistics will be reset after each run.

PrecisionRecall

class neoml.Dnn.PrecisionRecall(input_layers, reset=True, name=None)

The layer that calculates the number of objects classified correctly for either class in a binary classification scenario. Using these statistics, you can easily calculate the precision and recall for the trained network.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.

Layer inputs:

the network response. The dimensions:
- BatchLength * BatchWidth * ListSize is the number of objects classified
- Height, Width, Depth, Channels are 1
the correct class labels (1 or -1). The dimensions are the same as for the first input.

Layer outputs:

the 4-element array along the Channels dimension that contains:
- 0 - the number of correctly classified objects in class 1
- 1 - the total number of objects in class 1
- 2 - the number of correctly classified objects in class -1
- 3 - the total number of objects in class -1

property reset: Checks if the statistics will be reset after each run.

property result: Returns the result 4-element array, in the same order as the output blob.

ConfusionMatrix

class neoml.Dnn.ConfusionMatrix(input_layers, reset=True, name=None)

The layer that calculates the confusion matrix for classification results. The columns correspond to the network response, the rows - to the correct labels. Each element of the matrix contains the number of objects that belong to the “row” class and were classified as the “column” class.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
reset (bool, default=True) – Specifies if the statistics should be reset with each run. Set to False to accumulate statistics for subsequent runs.
name (str, default=None) – The layer name.

Layer inputs:

a blob with the network response The dimensions:
- BatchLength * BatchWidth * ListSize equal to the number of objects that were classified
- Height, Width, Depth equal to 1
- Channels equal to the number of classes and should be greater than 1
a blob with the correct class labels The dimensions should be the same as for the first input

Layer outputs:

the confusion matrix. The dimensions:
- BatchLength, BatchWidth, ListSize, Depth, Channels are 1
- Height and Width are equal to the input Channels

property matrix

Gets the confusion matrix. The dimensions:

BatchLength, BatchWidth, ListSize, Depth, Channels are 1
Height and Width are equal to the input Channels

property reset: Checks if the calculations will be reset on each run.

reset_matrix()

Resets the confusion matrix values. The dimensions:

BatchLength, BatchWidth, ListSize, Depth, Channels are 1
Height and Width are equal to the input Channels

Working with discrete features

AccumulativeLookup

class neoml.Dnn.AccumulativeLookup(input_layer, count, size, name=None)

The layer that trains fixed-length vector representations for the values of a discrete feature. It can work only with one feature. When several values of the feature are passed, the sum of the corresponding vectors is returned.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
count (int) – The number of vectors in the representation table.
size (int) – The length of each vector in the representation table.
name (str, default=None) – The layer name.

Layer inputs:

a data blob with integer data that contains the feature values. The dimensions:
- BatchLength * BatchWidth * ListSize equal to the number of different values the feature can take
- Height * Width * Depth * Channels equal to the number of values in the set

Layer outputs:

a blob with the sum of vector representations of the given feature values. The dimensions:
- BatchLength, BatchWidth, ListSize equal to these dimensions of the input
- Height, Width, Depth equal to 1
- Channels equal to the vector length (size parameter below)

property count: Gets the number of vectors.

property size: Gets the vector length.

MultichannelLookup

class neoml.Dnn.MultichannelLookup(input_layers, dimensions=None, name=None)

The layer that trains fixed-length vector representation for the values of several discrete features. See https://en.wikipedia.org/wiki/Word2vec, https://en.wikipedia.org/wiki/GloVe_(machine_learning)

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
dimensions (list of tuple(int, int)) – Each of the elements specifies the number and length of vectors in the representation table with the element’s index.
name (str, default=None) – The layer name.

Layer inputs:

a blob with feature values (float or int). The dimensions:
- BatchLength * BatchWidth * ListSize * Height * Width * Depth is the number of features
- Channels is the dimension along which the feature values for different sets are stored. Not smaller than the number of feature sets.

Layer outputs:

the blob with vectors for each input feature. The dimensions:
- BatchLength, BatchWidth, ListSize, Height, Width, Depth are the same as for the input
- Channels is the sum of vector lengths of all sets and additional channels if the input Channels is more than the number of tables.

property dimensions: Gets the list of representation table sizes.

get_embeddings(index)

Gets the representation table with the given index as a blob of the dimensions:

BatchLength * BatchWidth * ListSize is dimensions[i].VectorCount

Height * Width * Depth * Channels is dimensions[i].VectorSize

initialize(initializer): Specifies a different initializer for this layer than the one set for the whole network in general.

set_embeddings(index, blob)

Sets the representation table with the given index as a blob of the dimensions:

BatchLength * BatchWidth * ListSize is dimensions[i].VectorCount

Height * Width * Depth * Channels is dimensions[i].VectorSize

PositionalEmbedding

class neoml.Dnn.PositionalEmbedding(input_layer, type_name, name=None)

The layer that maps positions in sequence into vectors, optionally trainable. The exact formula depends on the type_name parameter.

The formula for “transformers”: result[i][j][k] = input[i][j][k] + sin(j / pow(10000, (k / vectorLength))), where:

i is the index of sequence in batch (from 0 to BatchWidth - 1)

j is the position of vector in sequence (from 0 to ListSize - 1)

k is the index of the element in the vector (from 0 to vectorLength - 1)

The formula for “learnable_addition”: result[i][j] = input[i][j] + addends[j], where:

i is the index of sequence in batch (from 0 to BatchWidth - 1)

j is the position of vector in sequence (from 0 to ListSize - 1)

addends is the trainable vector to be added

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
type_name (str, {"learnable_addition", "transformers"}) – The operation type.
name (str, default=None) – The layer name.

Layer inputs:

a blob with vector sequences. The dimensions:
- BatchLength is 1
- BatchWidth is the number of sequences in the set
- ListSize is the sequence length
- Height * Width * Depth * Channels is the vector length; for “transformers”, Height, Width, and Depth should be 1, vector length is equal to Channels

Layer outputs:

the transformation result, of the same dimensions as the input.

property addends

Gets the trainable vectors added. The blob dimensions:

BatchLength, BatchWidth are 1
the other dimensions are the same as for the input

property type: Gets the type of operation the layer performs.

TiedEmbeddings

class neoml.Dnn.TiedEmbeddings(input_layers, embeddings_layer_name, channel, name=None)

The tied embeddings layer. See https://arxiv.org/pdf/1608.05859.pdf The representations table is taken from a MultichannelLookup layer.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
embeddings_layer_name (str) – The name of the layer used for embeddings. Needs to be a MultichannelLookup layer.
channel (int, >=0) – The channel index in the embeddings layer.
name (str, default=None) – The layer name.

Layer inputs:

The layer may have any number of inputs, of the dimensions:

BatchLength * BatchWidth * ListSize is the number of objects
Height, Width, Depth are 1
Channels is the embedding size

Layer outputs:

For each input the layer has one output of the same dimensions.

property channel: Gets the channel index in the embeddings layer.

property embeddings_layer_name: Gets the name of the layer used for representation table.

EnumBinarization

class neoml.Dnn.EnumBinarization(input_layer, enum_size, name=None)

The layer that converts enumeration values into one-hot encoding.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
enum_size (int, > 0) – The number of constants in the enumeration.
name (str, default=None) – The layer name.

Layer inputs:

a blob with int or float data that contains enumeration values. The dimensions:
- Channels is 1
- the other dimensions may be of any length

Layer outputs:

a blob with the vectors that one-hot encode the enumeration values. The dimensions:
- Channels is enum_size
- the other dimensions stay the same as in the first input

property enum_size: Gets the number of constants in the enumeration.

BitSetVectorization

class neoml.Dnn.BitSetVectorization(input_layer, bit_set_size, name=None)

The layer that converts a bitset into vectors of ones and zeros.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
bit_set_size (int, > 0) – The size of the bitset.
name (str, default=None) – The layer name.

Layer inputs:

a blob with int data containing bitsets. The dimensions:
- BatchLength * BatchWidth * ListSize * Height * Width * Depth is the number of bitsets
- Channels is bitset itself

Layer outputs:

a blob with the result of vectorization. The dimensions:
- Channels is equal to bit_set_size
- the other dimensions are the same as for the input

property bit_set_size: Gets the bitset size.

Attention layers

AttentionDecoder

class neoml.Dnn.AttentionDecoder(input_layers, score, hidden_size, output_object_size, output_seq_len, name=None)

The layer that converts the input sequence into the output sequence, not necessarily of the same length

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
score (str, {'additive', 'dot_product'}) –
The type of estimate function used to check alignment of input and output sequences:
- additive is tanh(x*Wx + y*Wy)*v
- dot_product is x*W*y
hidden_size (int) – The size of the hidden layer.
output_object_size (int) – The number of channels in the output object.
output_seq_len (int) – The length of the output sequence.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size with the input sequence
the special character used to initialize the output sequence All dimensions are equal to 1

Layer outputs:

the output sequence The dimensions:
- BatchLength equal to output_seq_len
- Channels equal to output_object_size
- all other dimensions are equal to 1

property hidden_layer_size: Gets the size of the hidden layer.

property output_object_size: Gets the number of channels in the output.

property output_seq_len: Gets the length of the output sequence.

property score: Gets the estimate function.

MultiheadAttention

class neoml.Dnn.MultiheadAttention(input_layer, head_count, hidden_size, output_size, dropout_rate, name=None)

The multihead self-attention layer that transforms a set of matrices according to a formula:

\(Q = W_Q * Q\), \(K = W_K * K\), \(V = W_V * V\) where \(W_*\) are trainable matrices of size (Channels_* x GetHiddenSize())

\(Attention(Q, K, V) = softmax( Q * K_t / \sqrt{d_K} ) * V\) where \(d_k\) - dimension of k

\(MultiHeadAttention = dropout\_if\_needed(concat( head_1, ..., head_N )) * W_O\) where \(head_i = Attention( W_{Q,i} * X, W_{K,i} * X, W_{V,i} * X )\), \(W_*\) - trainable parameters and \(W_O\) is an additional trainable matrix of size (GetHiddenSize() x GetOutputSize())

See the papers: https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1807.03819.pdf

Parameters:

input_layer (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
head_count (int, > 0, default=1) – The number of heads in attention.
hidden_size (int, > 0) – The size of trainable matrices W_*. Should be a multiple of head_count.
output_size (int, > 0) – The size of output.
dropout_rate (float, default=-1) – Rate of droupout applied to the softmax. A negative value means no dropout.
name (str, default=None) – The layer name.

Layer inputs:

matrix Q of shape (1 x BatchWidth x ListSize_Q x 1 x 1 x 1 x Channels_Q)
matrix K of shape (1 x BatchWidth x ListSize_V x 1 x 1 x 1 x Channels_Q)
matrix V (1 x BatchWidth x ListSize_V x 1 x 1 x 1 x Channels_V)
(optional): mask, can be 0.

Layer outputs:

result matrix of shape (1, BatchWidth, ListSize_Q, 1, 1, 1, output_size)

property dropout_rate: Gets the rate of dropout applied to the softmax.

property head_count: Gets the number of heads in attention.

property hidden_size: Gets the trainable matrices size.

property output_size: Gets the output size.

property use_mask: Checks if the mask will be used.

Logical operations Layers

Not

class neoml.Dnn.Not(input_layer, name=None)

The layer that calculates logical not for each element of the single input:

\(f(x) = x == 0 ? 1 : 0\)

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a blob with integer data of any size

Layer outputs:

a blob with integer data of the same size with the result

Less

class neoml.Dnn.Less(input_layers, name=None)

The layer that compares 2 inputs element-by-element:

\(less[i] = first[i] < second[i] ? 1 : 0\)

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer must have 2 inputs of the same size and data type.

Layer outputs:

A blob with integer data of the same size as the inputs.

Equal

class neoml.Dnn.Equal(input_layers, name=None)

The layer that compares 2 inputs element-by-element:

\(equal[i] = first[i] == second[i] ? 1 : 0\)

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer must have 2 inputs of the same size and data type.

Layer outputs:

A blob with integer data of the same size as the inputs.

Where

class neoml.Dnn.Where(input_layers, name=None)

The layer that merges 2 blobs based on a mask:

\(where[i] = first[i] != 0 ? second[i] : third[i]\)

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer must have 3 inputs of the same size. First input must be of integer data type. Data types of second and third inputs must match.

Layer outputs:

A blob with integer data of the same size and data type as the second input.

Auxiliary operations

Transform

class neoml.Dnn.Transform(input_layer, transforms, name=None)

The layer that changes the input blob dimensions without moving any of the data. The total number of elements in the blob stays the same, and therefore the product of all dimensions should not be changed by the transformation.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
transforms (array of 7 tuples (operation, value), operation: one of "remainder", "set", "multiply", "divide" value: int > 0) –
Specifies the transformation to be made to each of the 7 dimensions:
- ”set” its length to the int value
- ”multiply” by the int value
- ”divide” by the int value
- ”remainder” may only be set for one dimension; it will be set so that the total size of the blob stays the same
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size.

Layer outputs:

a blob of the dimensions determined by the rules:
- the dimensions in “set” mode will be equal to the specified value
- the dimensions in “multiply” mode will be value times larger
- the dimensions in “divide” mode will be value times smaller
- the dimension in “remainder” mode will be such that the total size of the input and the output are the same

property transforms: Gets the array of transformations.

Transpose

class neoml.Dnn.Transpose(input_layer, first_dim='height', second_dim='width', name=None)

The layer that switches two of the blob dimensions, moving the data inside accordingly.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
first_dim (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – One of the dimensions that should be switched.
second_dim (str, {"batch_length", "batch_width", "list_size", "height", "width", "depth", "channels"}) – The other dimension that should be switched.
name (str, default=None) – The layer name.

Layer inputs:

a data blob to be transposed, of any size.

Layer outputs:

the result of transposition.

property first_dim: Gets the first dimension to be switched.

property second_dim: Gets the second dimension to be switched.

Argmax

class neoml.Dnn.Argmax(input_layer, dimension='channels', name=None)

The layer that finds the maximum element along the given dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
dimension (str, {'batch_length', 'batch_width', 'list_size', 'height', 'width', 'depth', 'channels'}, default='channels') – The dimension along which the maximum is to be found.
name (str, default=None) – The layer name.

Layer inputs:

a data blob of any size

Layer outputs:

contains the coordinate of the maximum The dimensions:
- ‘dimension’ along which the maximum is found is equal to 1
- all other dimensions are equal to the first input dimensions

property dimension: Gets the dimension along which the maximum is to be found.

DotProduct

class neoml.Dnn.DotProduct(input_layers, name=None)

The layer that calculates the dot product of its two inputs: each object in the first input is multiplied by the object with the same index in the second input.

Parameters:

input_layer (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer has two inputs, which must contain blobs of the same dimensions.

Layer outputs:

a blob with the dot product. The dimensions:
- BatchLength, BatchWidth, ListSize equal to the inputs’ dimensions
- Height, Width, Depth, Channels equal to 1

MatrixMultiplication

class neoml.Dnn.MatrixMultiplication(input_layers, name=None)

The layer that multiplies two sets of matrices.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

the first set of matrices. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of matrices in the set
- Height * Width * Depth - the height of each matrix
- Channels - the width of each matrix
the second set of matrices. The dimensions:
- BatchLength * BatchWidth * ListSize - the number of matrices in the set, must be the same as for the first input
- Height * Width * Depth - the height of each matrix, must be equal to Channels of the first input
- Channels - the width of each matrix

Layer outputs:

the set of multiplication results. The dimensions:
- BatchLength, BatchWidth, ListSize, Height, Width, Depth the same as for the first input
- Channels the same as for the second input

Reorg

class neoml.Dnn.Reorg(input_layer, stride=1, name=None)

The layer that transforms a set of two-dimensional multi-channel images into a set of images of smaller size but with more channels. This operation is used in YOLO architecture: https://pjreddie.com/darknet/yolo/

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
stride (int, default=1) – The value by which the image size will be divided in the final result.
name (str, default=None) – The layer name.

Layer inputs:

a blob with images. The dimensions:
- BatchLength * BatchWidth * ListSize is the number of images
- Height is the image height; should be a multiple of stride
- Width is the image width; should be a multiple of stride
- Depth is 1
- Channels is the number of channels the image format uses

Layer outputs:

the result of image transformation. The dimensions:
- BatchLength, BatchWidth, ListSize are the same as for the input
- Height is input Height / stride
- Width is input Width / stride
- Depth is 1
- Channels is input Channels * stride^2

property stride: Gets the divider value.

Concatenation layers

The layers that concatenate blobs along one of the dimensions.

ConcatChannels

class neoml.Dnn.ConcatChannels(input_layers, name=None)

The layer that concatenates several blobs into one along the Channels dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Width, Depth equal for all inputs
Channels dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Width, Depth equal to the inputs’ dimensions
Channels equal to the sum of all inputs’ Channels

ConcatDepth

class neoml.Dnn.ConcatDepth(input_layers, name=None)

The layer that concatenates several blobs into one along the Depth dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Width, Channels equal for all inputs
Depth dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Width, Channels equal to the inputs’ dimensions
Depth equal to the sum of all inputs’ Depth

ConcatWidth

class neoml.Dnn.ConcatWidth(input_layers, name=None)

The layer that concatenates several blobs into one along the Width dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Depth, Channels equal for all inputs
Width dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, ListSize, Height, Depth, Channels equal to the inputs’ dimensions
Width equal to the sum of all inputs’ Width

ConcatHeight

class neoml.Dnn.ConcatHeight(input_layers, name=None)

The layer that concatenates several blobs into one along the Height dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, ListSize, Width, Depth, Channels equal for all inputs
Height dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, ListSize, Width, Depth, Channels equal to the inputs’ dimensions
Height equal to the sum of all inputs’ Height

ConcatBatchWidth

class neoml.Dnn.ConcatBatchWidth(input_layers, name=None)

The layer that concatenates several blobs into one along the BatchWidth dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, ListSize, Height, Width, Depth, Channels equal for all inputs
BatchWidth dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, ListSize, Height, Width, Depth, Channels equal to the inputs’ dimensions
BatchWidth equal to the sum of all inputs’ BatchWidth

ConcatBatchLength

class neoml.Dnn.ConcatBatchLength(input_layers, name=None)

The layer that concatenates several blobs into one along the BatchLength dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchWidth, ListSize, Height, Width, Depth, Channels equal for all inputs
BatchLength dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchWidth, ListSize, Height, Width, Depth, Channels equal to the inputs’ dimensions
BatchLength equal to the sum of all inputs’ BatchLength

ConcatListSize

class neoml.Dnn.ConcatListSize(input_layers, name=None)

The layer that concatenates several blobs into one along the ListSize dimension.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, Height, Width, Depth, Channels equal for all inputs
ListSize dimension may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, Height, Width, Depth, Channels equal to the inputs’ dimensions
ListSize equal to the sum of all inputs’ ListSize

ConcatObject

class neoml.Dnn.ConcatObject(input_layers, name=None)

The layer that concatenates several blobs into one along the Height, Width, Depth, and Channels dimensions.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer accepts an arbitrary number of inputs. The dimensions:

BatchLength, BatchWidth, ListSize equal for all inputs
Height, Width, Depth, Channels dimensions may vary

Layer outputs:

a blob with the result of concatenation. The dimensions:

BatchLength, BatchWidth, ListSize equal to the inputs’ dimensions
Height, Width, Depth equal to 1
Channels equal to the sum of Height * Width * Depth * Channels over all inputs

Split layers

The layers that split a blob along one of the dimensions.

SplitChannels

class neoml.Dnn.SplitChannels(input_layer, sizes, name=None)

The layer that splits an input blob along the Channels dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- Channels should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

Channels equals the corresponding element of sizes array, for the last output it is input Channels minus the sum of sizes
all other dimensions are the same as for the input

SplitDepth

class neoml.Dnn.SplitDepth(input_layer, sizes, name=None)

The layer that splits an input blob along the Depth dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- Depth should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

Depth equals the corresponding element of sizes array, for the last output it is input Depth minus the sum of sizes

all other dimensions are the same as for the input

SplitWidth

class neoml.Dnn.SplitWidth(input_layer, sizes, name=None)

The layer that splits an input blob along the Width dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- Width should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

Width equals the corresponding element of sizes array, for the last output it is input Width minus the sum of sizes

all other dimensions are the same as for the input

SplitHeight

class neoml.Dnn.SplitHeight(input_layer, sizes, name=None)

The layer that splits an input blob along the Height dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- Height should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

Height equals the corresponding element of sizes array, for the last output it is input Height minus the sum of sizes

all other dimensions are the same as for the input

SplitListSize

class neoml.Dnn.SplitListSize(input_layer, sizes, name=None)

The layer that splits an input blob along the ListSize dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- ListSize should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

ListSize equals the corresponding element of sizes array, for the last output it is input ListSize minus the sum of sizes

all other dimensions are the same as for the input

SplitBatchWidth

class neoml.Dnn.SplitBatchWidth(input_layer, sizes, name=None)

The layer that splits an input blob along the BatchWidth dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- BatchWidth should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

BatchWidth equals the corresponding element of sizes array, for the last output it is input BatchWidth minus the sum of sizes

all other dimensions are the same as for the input

SplitBatchLength

class neoml.Dnn.SplitBatchLength(input_layer, sizes, name=None)

The layer that splits an input blob along the BatchLength dimension.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
sizes (array of int, up to 3 elements) – The sizes of the first one, two, or three parts. The final part size is what’s left.
name (str, default=None) – The layer name.

Layer inputs:

a blob with input data. The dimensions:
- BatchLength should not be less than the sum of sizes array elements.

Layer outputs:

The layer has at least len(sizes) outputs. The dimensions:

BatchLength equals the corresponding element of sizes array, for the last output it is input BatchLength minus the sum of sizes

all other dimensions are the same as for the input

Working with sequences

SubSequence

class neoml.Dnn.SubSequence(input_layer, start_pos=0, length=1, name=None)

The layer that extracts a subsequence from each vector sequence of the set.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
start_pos (int) – The first element of the subsequence. Counted from the end if negative.
length (int) – The length of the subsequence. Reversed order if negative.
name (str, default=None) – The layer name.

Layer inputs:

a blob with a set of objects, numbered along the BatchWidth dimension.

Layer outputs:

a blob with the subsequence of objects. The dimensions:
- BatchWidth is abs(length) or smaller if it doesn’t fit after starting at start_pos
- the other dimensions are the same as for the input

property length: Gets the subsequence length.

property start_pos: Gets the starting position.

ReverseSequence

class neoml.Dnn.ReverseSequence(input_layer, name=None)

The layer that reverses sequence order of the input.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a blob of any size with sequence of objects numbered along the BatchWidth dimension.

Layer outputs:

the reverse sequence. The same size as the input.

SequenceSum

class neoml.Dnn.SequenceSum(input_layer, name=None)

The layer that adds up object sequences.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a set of object sequences. The dimensions:
- BatchLength is the sequence length
- BatchWidth is the number of sequences in the set
- ListSize is 1
- Height * Width * Depth * Channels is the object size

Layer outputs:

the results of adding up each of the sequences. The dimensions:
- BatchLength is 1
- the other dimensions are the same as for the input

Image conversion layers

ImageResize

class neoml.Dnn.ImageResize(input_layer, deltas, default_value=0.0, name=None)

The layer that resizes a set of two-dimensional multi-channel images.

Parameters:

input_layer (object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
deltas (str, ("left", "right", "top", "bottom")) – The differences between the original and the resized image, on each side. If the difference is negative, rows or columns are removed from the specified side. If it is positive, rows or columns are added and filled with the default_value pixels.
default_value (float, default=0.0) – The value for the added pixels.
name (str, default=None) – The layer name.

Layer inputs:

a set of images, of the dimensions:
- BatchLength * BatchWidth * ListSize - the number of images
- Height - the images’ height
- Width - the images’ width
- Depth * Channels - the number of channels the image format uses

Layer outputs:

a blob with the resized images, of the dimensions:
- BatchLength, BatchWidth, ListSize, Depth, Channels are equal to the input dimensions
- Height is the input Height plus the sum of top and bottom deltas
- Width is the input Width plus the sum of right and left deltas

property default_value: Gets the default value for new pixels.

property deltas: Gets the size differences on each side.

PixelToImage

class neoml.Dnn.PixelToImage(input_layer, height, width, name=None)

The layer that creates a set of two-dimensional images using a set of pixel sequences with specified coordinates.

Parameters:

input_layer (list of object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
height (int) – The height of the resulting images.
width (int) – The width of the resulting images.
name (str, default=None) – The layer name.

Layer inputs:

a blob with pixel sequences. The dimensions:
- BatchLength is 1
- BatchWidth is the number of sequences in the set
- ListSize is the length of each sequence
- Height, Width, Depth are 1
- Channels is the number of channels for the pixel sequences and the output images.
a blob with integer data that contains lists of pixel coordinates. The dimensions:
- BatchWidth, ListSize are the same as for the first input
- the other dimensions are 1

Layer outputs:

a blob with images. The dimensions:
- BatchLength is 1
- BatchWidth is the same as for the first input
- ListSize is 1
- Height is the specified image height
- Width is the specified image width
- Depth is 1
- Channels is the same as for the first input

property height: Gets the output image height.

property width: Gets the output image width.

ImageToPixel

class neoml.Dnn.ImageToPixel(input_layer, name=None)

The layer that extracts a set of pixel sequences along the specified coordinates from a set of two-dimensional images.

Parameters:

input_layer (list of object, tuple(object, int)) – The input layer and the number of its output. If no number is specified, the first output will be connected.
name (str, default=None) – The layer name.

Layer inputs:

a set of two-dimensional images. The blob dimensions:
- BatchLength is 1
- BatchWidth is the number of sequences in the set
- ListSize 1
- Height is the images’ height
- Width is the images’ width
- Depth is 1
- Channels is the number of channels the image format uses
a blob with integer data that contains the pixel sequences. The dimensions:
- BatchWidth is the same as for the first input
- ListSize is the length of each sequence
- all other dimensions are 1

Layer outputs:

a blob with the pixel sequences. The dimensions:
- BatchLength is 1
- BatchWidth is the inputs’ BatchWidth
- ListSize is the same as for the second input
- Height, Width, Depth are 1
- Channels is the same as for the first input

Upsampling2D

class neoml.Dnn.Upsampling2D(input_layers, height_copy_count, width_copy_count, name=None)

The layer that scales up a set of two-dimensional multi-channel images. The new pixels are filled up by repeating the existing pixels’ values, without any interpolation.

Parameters:

input_layers (object, tuple(object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
height_copy_count (int, > 0) – The height multiplier.
width_copy_count (int, > 0) – The width multiplier.
name (str, default=None) – The layer name.

Layer inputs:

The layer can have any amount of inputs, each with a set of images. The dimensions:

BatchLength * BatchWidth * ListSize is the number of images
Height is the images’ height
Width is the images’ width
Depth * Channels is the number of channels the image format uses

Layer outputs:

For each input, there is a corresponding output with upscaled images. The dimensions:

BatchLength, BatchWidth, ListSize, Depth, Channels equal the correspoonding input dimensions
Height is height_copy_count times larger than the input Height
Width is width_copy_count times larger than the input Width

property height_copy_count: Gets the height multiplier.

property width_copy_count: Gets the width multiplier.

Scatter & Gather Layers

ScatterND

class neoml.Dnn.ScatterND(input_layers, name=None)

The layer that updates certain objects in data:

\(data[indices[i]] = updates[i]\)

where indices[…] is an integer vector of IndexDims elements containing coordinates in the first IndexDims dimensions of the data blob

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

The layer must have 3 inputs.

1. Data. Set of objects of any type. The product of its first IndexDims dimensions is ObjectCount. The product of the rest of the dimensions is ObjectSize.

2. Indices. Blob of integer data type. The number of channels of this blob is IndexDims. The total size must be UpdateCount * IndexDims.

3. Updates. Blob of the same data type as the first. The total size of the blob is UpdateCount * ObjectSize.

Layer outputs:

A blob of the same data type and size as first input.

Conditional random field (CRF)

Conditional random field is implemented in three parts:

the trainable layer that contains the transition probabilities of the random field
the loss function optimized by training
the special layer to extract highest-probability sequences from the output of the trained CRF layer

Crf

class neoml.Dnn.Crf(input_layer, class_count, padding=0, dropout_rate=0.0, name=None)

The layer that trains and calculates transitional probabilities in a conditional random field (CRF).

Parameters:

input_layer (object, tuple(object, int) of list of them) – The input layer and the number of its output. If no number is specified, the first output will be connected.
class_count (int) – The number of classes in the CRF.
padding (int, default=0) – The number of empty class used to fill the sequence end.
dropout_rate (float, default=0.0) – Variational dropout.
name (str, default=None) – The layer name.

Layer inputs:

a blob with the object sequences. The dimensions:
- BatchLength is the sequence length
- BatchWidth is the number of sequences in the set
- ListSize is 1
- Height * Width * Depth * Channels is the object size
(optional): a blob with integer data that contains the correct class sequences. It is required only for training. The dimensions:
- BatchLength, BatchWidth equal to the first input’s
- the other dimensions are equal to 1

Layer outputs:

(optional): a blob with integer data that contains optimal class sequences. This output will be returned only if you set calc_prev_best_class to True. During training, this output usually isn’t needed and is switched off by default. The dimensions:
- BatchLength, BatchWidth equal to the first input’s
- Channels equal to the number of classes
- the other dimensions are equal to 1
a blob with float data that contains non-normalized logarithm of optimal class sequences probabilities. The dimensions are the same as for the first output.
(optional): a blob with non-normalized logarithm of the correct class sequences probabilities. This output will be there only if the layer has two inputs. The dimensions are equal to the second input’s:
- BatchLength, BatchWidth equal to the first input’s
- the other dimensions are equal to 1

property calc_best_prev_class: Checks if the first output will be returned.

property class_count: Gets the number of classes in the CRF.

property dropout_rate: Gets the variational dropout rate.

property free_terms: Gets the hidden layer free terms. The blob size is class_count.

property hidden_weights

Gets the hidden layer weights. The dimensions:

BatchLength * BatchWidth * ListSize is the number of classes
Height * Width * Depth * Channels the same as for the first input

property padding: Gets the number of the empty class.

property transitions

Gets the transition probability matrix. The dimensions:

BatchLength, BatchWidth are class_count
the other dimensions are 1

CrfLoss

class neoml.Dnn.CrfLoss(input_layers, loss_weight=1.0, name=None)

The layer that calculates the loss function used for training a CRF. The value is -log(probability of the correct class sequence)

Parameters:

input_layers (array of (object, int) tuples or objects) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

a blob with integer data that contains optimal class sequences. The dimensions: - BatchLength, BatchWidth equal to the network inputs’ - Channels equal to the number of classes - the other dimensions are equal to 1
a blob with float data containing non-normalized logarithm of probabilities of the optimal class sequences. The dimensions are the same as for the first input.
a blob with float data containing non-normalized logarithm of probability of the correct class being in this position. The dimensions:
- BatchLength, BatchWidth the same as for the first input
- the other dimensions equal to 1

Layer outputs:

The layer has no output.

property last_loss: Gets the value of the loss function on the last step.

property loss_weight: Gets the multiplier for the loss function value dduring training.

property max_gradient: Gets the gradient clipping threshold.

BestSequence

class neoml.Dnn.BestSequence(input_layers, name=None)

The layer that finds the optimal class sequence using the Crf layer output.

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
name (str, default=None) – The layer name.

Layer inputs:

first output of Crf. A blob with int data that contains the optimal class sequences. The dimensions:
- BatchLength is the sequence length
- BatchWidth is the number of sequences in the set
- Channels is the number of classes
- all other dimensions are 1
second output of Crf. A blob with float data that contains non-normalized logarithm of optimal class sequences probabilities. The dimensions are the same as for the first input.

Layer outputs:

a blob with int data that contains the optimal class sequence. The dimensions:
- BatchLength, BatchWidth are the same as for the inputs
- the other dimensions are 1

Connectionist temporal classification (CTC)

A connectionist temporal classification (CTC) network is trained to optimize the loss function.

After training, use the special decoding layer to extract the optimal sequences from the network output.

CtcLoss

class neoml.Dnn.CtcLoss(input_layers, blank, skip, loss_weight=1.0, name=None)

The layer that calculates the loss function used for connectionist temporal classification (CTC). See https://www.cs.toronto.edu/~graves/preprint.pdf

Parameters:

input_layers (list of object, tuple(object, int)) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
blank (int) – Sets the value for the blank label that will be used as space.
skip (bool) – Specifies if blank labels may be skipped when aligning.
loss_weight (float, default=1.0) – The multiplier for the loss function value during training.
name (str, default=None) – The layer name.

Layer inputs:

the network response. The dimensions:
- BatchLength is the maximum sequence length
- BatchWidth is the number of sequences in the set
- ListSize is 1
- Height * Width * Depth * Channels is the number of classes
the correct labels as a blob with int data.

The dimensions: - BatchLength is the maximum labels sequence length - BatchWidth is the number of sequences, same as first input’s BatchWidth - the other dimensions are 1
(optional): the label sequences lengths as a blob with int data. If this input isn’t connected, the label sequences are considered to be the second input’s BatchLength long. The dimensions:
- BatchWidth is the same as for the first input
- the other dimensions are 1
(optional): the network response sequences lengths as a blob with int data. If this input isn’t connected, the response sequences are considered to be the first input’s BatchLength long. The dimensions:
- BatchWidth is the same as for the first input
- the other dimensions are 1
(optional): the sequences’ weights. The dimensions:
- BatchWidth is the same as for the first input
- the other dimensions are 1

Layer outputs:

The layer has no output.

property blank: Gets the value of the blank label.

property last_loss: Gets the value of the loss function on the last step.

property loss_weight: Gets the multiplier for the loss function during training.

property max_gradient: Gets the upper limit for the absolute value of the function gradient.

property skip: Checks if blank labels may be skipped when aligning.

CtcDecoding

class neoml.Dnn.CtcDecoding(input_layers, blank, blank_threshold, arc_threshold, name=None)

The layer that is looking for the most probable sequences in the response of a connectionist temporal classification (CTC) network.

Parameters:

input_layers (object, (object, int) or list of them) – The input layers to be connected. The integer in each tuple specifies the number of the output. If not set, the first output will be used.
blank (int) – Sets the value for the blank label that will be used as space.
blank_threshold (float, [0..1]) – The probability threshold for blank labels when building a linear division graph (LDG).
arc_threshold (float, [0..1]) – The probability threshold for cutting off arcs when building a linear division graph (LDG).
name (str, default=None) – The layer name.

Layer inputs:

the network response. The dimensions:
- BatchLength is the maximum sequence length
- BatchWidth is the number of sequences in the set
- ListSize is 1
- Height * Width * Depth * Channels is the number of classes
(optional): the network response sequences lengths as a blob with int data. If this input isn’t connected, the response sequences are considered to be the first input’s BatchLength long. The dimensions:
- BatchWidth is the same as for the first input
- the other dimensions are 1

Layer outputs:

The layer has no output.

property arc_threshold: Gets the probability threshold for cutting off arcs when building a linear division graph (LDG).

property batch_width: Returns the number of sequences.

property blank: Gets the value of the blank label.

property blank_threshold: Gets the probability threshold for blank layers when building a linear division graph (LDG).

get_best_sequence(sequence_number): Retrieves the most probable sequence for the object with sequence_number index in the set.

property label_count: Returns the number of classes.

property sequence_length: Returns the sequence length.

Initializers

Before the first training iteration the layers’ weights (trainable parameters) must be initialized. The initializer is the same for all the network trainable weights, except for the free term vectors that are initialized with zeros.

Xavier

The default initializer. It generates the weights using the normal distribution N(0, 1/n), where n is the input size.

class neoml.Dnn.Xavier(random_generator=None)

Initializes a blob using the Xavier algorithm: with random values from a normal distribution over (0, 1 / input_size).

Parameters:: random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.

XavierUniform

The default initializer. It generates the weights using the uniform distribution U(-sqrt(1/n), sqrt(1/n)), where n is the input size.

class neoml.Dnn.XavierUniform(random_generator=None)

Initializes a blob using the Xavier algorithm: with random values from a uniform distribution over (-sqrt(1 / input_size), sqrt(1 / input_size)).

Parameters:: random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.

Uniform

Generates the weights using a uniform distribution over the specified segment.

class neoml.Dnn.Uniform(lower_bound=-1.0, upper_bound=1.0, random_generator=None)

Initializes a blob using uniform distribution between the set bounds.

Parameters:

lower_bound (float, default=-1.0) – The lower bound of the distribution.
upper_bound (float, default=1.0) – The upper bound of the distribution.
random_generator (object, default=None) – Sets the random numbers generator to be used. By default, the standard NeoML randomizer with default seed is used.

property lower_bound: Gets the distribution lower bound.

property upper_bound: Gets the distibution upper bound.

Solvers

The optimizer, or solver, sets the rules to update the weights during training.

SimpleGradient

class neoml.Dnn.SimpleGradient(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=-1.0, moment_decay_rate=0.9)

Stochastic gradient descent with moment.

Parameters:

math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=-1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.

property moment_decay_rate: Gets the moment decay rate. Moment is a weighted sum of previous gradients.

AdaptiveGradient

class neoml.Dnn.AdaptiveGradient(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=-1.0, moment_decay_rate=0.9, second_moment_decay_rate=0.99, epsilon=1e-06, ams_grad=False, decoupled_weight_decay=False)

Gradient descent with adaptive momentum (Adam).

Parameters:

math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=-1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.
second_moment_decay_rate (float, default=0.99) – The decay rate for the weighted sum of previous gradients, squared, also called the second moment.
epsilon (float, default=1e-6) – The small value used to avoid division by zero when calculating second moment.
ams_grad (bool, default=False) – Turns AMSGrad mode on or off. AMSGrad helps against divergence and rapid vanishing of previous states memory, which may become a problem for the optimizers that use the moving mean for squared gradient history (Adam, NAdam, RMSprop). See https://openreview.net/pdf?id=ryQu7f-RZ.
decoupled_weight_decay (bool, default=False) – Enables using of Decoupled Weight Decay Regularization. See https://openreview.net/pdf?id=Bkg6RiCqY7.

property ams_grad: Checks if AMSGrad mode is on.

property decoupled_weight_decay: Checks if Decoupled Weight Decay Regularization is used.

property epsilon: Gets the small value used to avoid division by zero when calculating second moment.

property moment_decay_rate: Gets the moment decay rate. Moment is a weighted sum of previous gradients.

property second_moment_decay_rate: Gets the decay rate for the weighted sum of previous gradients, squared - that is, the second moment.

NesterovGradient

class neoml.Dnn.NesterovGradient(math_engine, learning_rate=0.01, l1=0, l2=0, max_gradient_norm=-1.0, moment_decay_rate=0.9, second_moment_decay_rate=0.99, epsilon=1e-06, ams_grad=False, decoupled_weight_decay=False)

The optimizer that uses Nesterov moment. See http://cs229.stanford.edu/proj2015/054_report.pdf (Algo 8).

Parameters:

math_engine (object) – The math engine to be used for calculations.
learning_rate (float, default=0.01) – The learning rate.
l1 (float, default=0) – The L1 regularization parameter.
l2 (float, default=0) – The L2 regularization parameter.
max_gradient_norm (float, default=-1.0) – The upper limit for gradient norm. A negative value means no limit, which is also the default setting.
moment_decay_rate (float, default=0.9) – The moment decay rate. Moment is a weighted sum of previous gradients.
second_moment_decay_rate (float, default=0.99) – The decay rate for the weighted sum of previous gradients, squared, also called the second moment.
epsilon (float, default=1e-6) – The small value used to avoid division by zero when calculating second moment.
ams_grad (bool, default=False) – Turns AMSGrad mode on or off. AMSGrad helps against divergence and rapid vanishing of previous states memory, which may become a problem for the optimizers that use the moving mean for squared gradient history (Adam, NAdam, RMSprop). See https://openreview.net/pdf?id=ryQu7f-RZ.
decoupled_weight_decay (bool, default=False) – Enables using of Decoupled Weight Decay Regularization. See https://openreview.net/pdf?id=Bkg6RiCqY7.

property ams_grad: Checks if AMSGrad mode is on.

property decoupled_weight_decay: Checks if Decoupled Weight Decay Regularization is used.

property epsilon: Gets the small value used to avoid division by zero when calculating second moment.

property moment_decay_rate: Gets the moment decay rate. Moment is a weighted sum of previous gradients.

property second_moment_decay_rate: Gets the decay rate for the weighted sum of previous gradients, squared - that is, the second moment.

Random

class neoml.Random.Random(*args: Any, **kwargs: Any)

The random numbers generator.

Parameters:: seed (int, default=None) – the seed of the random numbers generator.

Autodifferentiation

NeoML supports autodifferentiation for a wide set of operations. Use these operations and simple arithmetic if you’d like to create your own loss function neoml.Dnn.CustomLoss. Then during the backward pass, NeoML will be able to calculate gradients of your custom loss.

AutoDiff.const(shape, data): Creates a blob of the specified shape filled with data value.

Simple arithmetic operations

AutoDiff.add(b): Elementwise adds two blobs or a blob and a scalar value.

AutoDiff.sub(b): Elementwise subtracts two blobs or a blob and a scalar value.

AutoDiff.mul(b): Elementwise multiplies two blobs or a blob and a scalar value.

AutoDiff.div(b): Elementwise divides two blobs or a blob and a scalar value.

AutoDiff.less(b): Compare blobs elementwise.

Basic math functions

AutoDiff.max(b): Takes the elementwise maximum of a blob and a scalar value.

AutoDiff.sum(axes=None): Calculates sum of blob elements along provided axes. If axes=None calculates the total sum.

AutoDiff.cumsum(axis=0): Calculates cumulative sum of blob elements along provided axis.

AutoDiff.mean(axes=None): Calculates mean of blob elements along provided axes. If axes=None calculates the total mean.

AutoDiff.neg(): Returns the negative of a blob or a number.

AutoDiff.abs(): Takes absolute value of each blob element.

AutoDiff.log(): Takes the logarithm of each blob element.

AutoDiff.exp(): Takes the exponential of each blob element.

AutoDiff.pow(b): Computes the power of one blob to another elementwise.

Other operations

AutoDiff.concat(axis=0): Merges the blobs along given axis.

AutoDiff.reshape(shape): Reshape the blob.

AutoDiff.clip(min_value, max_value): Clips each element of the blob so that it fits between the specified limits.

AutoDiff.top_k(k=1): Finds values of the k largest elements in the blob. The result is a blob of size k.

AutoDiff.binary_cross_entropy(preds, fromLogits): Calculates binary cross-entropy for two blobs: the first one contains labels, the second one contains predictions. Blobs should be of the same shape. \(result = (1 - labels) * x + log(1 + exp(-x))\) if fromLogits then x = preds, else \(x = log( clippedPreds / (1 - clippedPreds))\)