README

CAI NEURAL API

Everyone says Pascal died in the 90s. Meanwhile, this Pascal library runs GPT-style transformers, Mixture-of-Experts, RWKV, xLSTM, spiking neurons, and normalizing flows, natively, with no Python nor CUDA lock-in. Your PyTorch project: 4GB of dependencies, a conda environment that breaks monthly. This: compile to a single native binary with AVX. This is deep learning the way systems programmers wish it worked.

And it actually runs the real thing. Point the ChatTerminal example at a HuggingFace instruct model — Qwen2.5 (0.5B up to 32B), Qwen3, Qwen3.8, Llama (TinyLlama, SmolLM2), Mistral 7B, Phi-3-mini or the OLMoE Mixture-of-Experts — and chat with a genuine LLM in your terminal, or expose it as an OpenAI-compatible server with ChatServer: OpenCL-accelerated, no Python, no CUDA — with more architectures landing all the time. Then hand a text prompt to MusicGenText and watch the same library turn "lo-fi hip-hop with a mellow piano" into actual audio — T5 encoder and MusicGen decoder. One native binary, from a 20-line CIFAR demo to a billion-parameter transformer.

CAI NEURAL API is a Pascal deep-learning library optimized for the AVX, AVX2 and AVX512 instruction sets and for OpenCL devices from AMD, Intel and NVIDIA. Tested on Windows and Linux.

Highlights

• 🗣️ Run real LLMs, natively — import a HuggingFace instruct checkpoint (Qwen2.5/Qwen3/Qwen3.8, Llama, Mistral, Phi-3, OLMoE) and chat from your terminal with ChatTerminal, or serve the same checkpoint over an OpenAI-style HTTP endpoint with ChatServer, so any OpenAI client or SDK can call it. ChatServer's POST /v1/systemone (TypeSafe/Jev typed decisions: a state and typed questions in, option probabilities out, through the AnyJev L0 readout, or through a Kev / Open-Jev LoRA checkpoint with --decision-adapter) supports Open-Jev-2B (user-tested on its Qwen/Qwen3.5-2B base with --int4 --gpu); AnyJev L0, Kev and the other Open-Jev sizes are planned (coded). No Python, no CUDA.
• 🎨 Text-to-image — turn a text prompt into a 1024x1024 image with QwenImage (Qwen-Image-2.1: Qwen3-VL text encoder, 7B MMDiT transformer, VAE decoder; int8/int4 weights, OpenCL GPU by default).
• 🎵 Text-to-music — turn a text prompt into audio with MusicGenText (T5 encoder + MusicGen decoder).
• 🔊 More audio — separate a song into stems with MusicSourceSeparation, talk to an audio-aware model with Qwen2AudioChat, or turn English text into speech with KokoroTTS (Kokoro-82M; a pure Pascal G2P reads misaki's dictionaries; REPL or one-shot).
• 🧠 Modern architectures, built in — GPT-style transformers, Mixture-of-Experts, RWKV, xLSTM, spiking neurons and normalizing flows, all as native Pascal layers.
• 🖼️ Image super-resolution — upscale images with a trained network via the SuperResolution command-line tool (ships with a ready-to-use model).
• ⚡ Fast everywhere — AVX/AVX2/AVX512 CPU paths plus OpenCL offload for AMD, Intel and NVIDIA GPUs.
• 📦 Single-binary deploy — compile to one self-contained native executable. No runtime, no dependency tree.
• 📖 Readable at native speed — clean, easy-to-learn Pascal that compiles to fast native code. No trade-off between readable source and raw performance.

Prerequisites

You'll need Lazarus development environment. If you have an OpenCL capable device, you'll need its OpenCL drivers. Many examples use the CIFAR-10 dataset. You'll also find examples for the CIFAR-100, MNIST, Fashion MNIST and the Places365-Standard Small images 256x256 dataset.

Will It Work with Delphi?

This project is Lazarus / Free Pascal based. The master branch does not work with Delphi. For Delphi, use v2.0.0 instead.

Installation

Clone this project, add the neural folder to your Lazarus unit search path and you'll be ready to go!

Documentation

The documentation covers:

• Easy examples
• Simple image classification examples
• Youtube videos
• Advanced examples
• Data structures (Volumes)
• Neural network layers
• Dataset support
• Training (fitting) your neural network
• Parallel computing
• Full set of examples
• Normalization Cheat Sheet
• Layer Authoring Guide — checklist for adding a new layer plus mini-guides on reading numerical-gradient failures and picking a tolerance
• Other scientific publications from the same author

Intro Videos

Basics of Neural Networks in Pascal - Loading and Saving	Neural Networks for Absolute Beginners! Learning a Simple Function	Coding a Neural Network in Pascal that Learns to Calculate the Hypotenuse

Easy Examples First Please!

You can click on the image above to watch the video.

Assuming that you would like to train a neural network to learn a function that has 2 inputs and one output, you could start with something like this:

    NN.AddLayer([
      TNNetInput.Create(2),
      TNNetFullConnectReLU.Create(32),
      TNNetFullConnectReLU.Create(32),
      TNNetFullConnectLinear.Create(1)
    ]);

The example above has 2 inputs (TNNetInput), 2 dense layers (TNNetFullConnectReLU) with 32 neurons each and one output (TNNetFullConnectLinear).

You can learn more about how to build and train simple neural networks at the following source code examples:

• Only one neuron
• Training a neural network to learn the hypotenuse function
• Training a neural network to learn the hypotenuse function with FitLoading
• Training a neural network to learn boolean functions AND, OR and XOR with neuralfit unit
• Training a neural network to learn boolean functions AND, OR and XOR without neuralfit unit
• Reptile first-order meta-learning: learning an initialization that adapts to a new sine-regression task in a few SGD steps
• Lottery Ticket Hypothesis: magnitude-prune a small dense net, then retrain the sparse mask from the original init vs from fresh random weights — the original-init "winning ticket" matches the dense net and beats random reinit at moderate-to-high sparsity (pure CPU)

Loading and Saving Neural Networks

Loading is very easy:

    NN := TNNet.Create;
    NN.LoadFromFile('MyTrainedNeuralNetwork.nn');

Saving is as easy:

    NN.SaveToFile('MyTrainedNeuralNetwork.nn');

Simple Image Classification Examples

CIFAR-10 Image Classification Example

The CIFAR-10 dataset is a well-known collection of images commonly used to train machine learning and computer vision algorithms. It was created by the Canadian Institute for Advanced Research (CIFAR). It contains 60K 32x32 color images. The images are classified into 10 different classes, with 6,000 images per class. The classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Despite its relatively low resolution and small size, CIFAR-10 can be challenging for models to achieve high accuracy, making it a good dataset for testing advancements in machine learning techniques.

Follows a source code example for the CIFAR-10 image classification:

NN := TNNet.Create();
NN.AddLayer([
  TNNetInput.Create(32, 32, 3), //32x32x3 Input Image
  TNNetConvolutionReLU.Create({Features=}16, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
  TNNetMaxPool.Create({Size=}2),
  TNNetConvolutionReLU.Create({Features=}32, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
  TNNetMaxPool.Create({Size=}2),
  TNNetConvolutionReLU.Create({Features=}32, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
  TNNetFullConnectReLU.Create({Neurons=}32),
  TNNetFullConnectLinear.Create(NumClasses),
  TNNetSoftMax.Create()
]);

CreateCifar10Volumes(ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes);

WriteLn('Neural Network will minimize error with:');
WriteLn(' Layers: ', NN.CountLayers());
WriteLn(' Neurons:', NN.CountNeurons());
WriteLn(' Weights:', NN.CountWeights());

NeuralFit := TNeuralImageFit.Create;
NeuralFit.InitialLearningRate := fLearningRate;
NeuralFit.Inertia := fInertia;
NeuralFit.Fit(NN, ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, NumClasses, {batchsize}128, {epochs}100);

These examples train a neural network to classify images in classes such as: image has a cat, image has a dog, image has an airplane...

• Simple CIFAR-10 Image Classifier
• Simple CIFAR-10 Image Classifier with OpenCL
• Many neural network architectures for CIFAR-10 image classification
• MNIST, Fashion MNIST and CIFAR-100

You can save and load trained models (neural networks) with TNNet.SaveToFile and TNNet.LoadFromFile. The file format is portable meaning that you can train on CPU and run on GPU or train in AMD and run on ARM as examples. The following code shows a simple example for image classification loading a pre-trained model:

  procedure ClassifyOneImageSimple;
  var
    NN: TNNet;
    ImageFileName: string;
    NeuralFit: TNeuralImageFit;
  begin
    WriteLn('Loading Neural Network...');
    NN := TNNet.Create;
    NN.LoadFromFile('SimplePlantLeafDisease-20230720.nn');
    NeuralFit := TNeuralImageFit.Create;
    ImageFileName := 'plant/Apple___Black_rot/image (1).JPG';
    WriteLn('Processing image: ', ImageFileName);
    WriteLn(
      'The class of the image is: ',
      NeuralFit.ClassifyImageFromFile(NN, ImageFileName)
    );
    NeuralFit.Free;
    NN.Free;
  end;

Youtube Videos

Basics of Neural Networks in Pascal - Loading and Saving	Neural Networks for Absolute Beginners! Learning a Simple Function	Coding a Neural Network in Pascal that Learns to Calculate the Hypotenuse
		https://youtu.be/o-8NuoSsdck
Pre-trained Neural Networks & Transfer Learning with Pascal's CAI Neural API	Coding a Neural Network in Pascal that Learns the OR Boolean Operation	A Dive into Identity Shortcut Connection - The ResNet building block
https://youtu.be/SEvWB7k8uy0	https://youtu.be/3QwIaAsDmJw	https://youtu.be/VH6v3D5cxxs
Increasing Image Resolution with Neural Networks	Ultra Fast Single Precision Floating Point Computing	AVX and AVX2 Code Optimization

Some videos make referrence to uvolume unit. The current neuralvolume unit used to be called uvolume. This is why it's mentioned.

Advanced Examples

Although these examples require deeper understanding about neural networks, they are very interesting:

• Identity Shortcut Connection - ResNet building block
• ResNet-20 - includes a web server example
• DenseNetBC L40
• Separable Convolutions - MobileNet building block
• Gradient Ascent - Visualizing patterns from inner neurons in image classification
• Artificial Art - Let a neural network produce art via a generative adversarial network
• Super Resolution - A neural network learns how to increase image resolution
• CIFAR-10 Resized - A program that resizes CIFAR-10 and CIFAR-100 images to 64x64 and 128x128 pixels.
• Autoencoder - Shows an autoencoder built with hyperbolic tangents and trained with Tiny ImageNet 200.

There is also a full set of examples that you can look at.

Volumes

Volumes behave like dynamically created arrays. They are the main array like structure used by this API. TNNetVolume class allows you to create volumes that can be accessed as 1D, 2D or 3D arrays and be operated with Advanced Vector Extensions (AVX) - Single Instruction Multiple Data (SIMD) instruction set. The usual way to create a volume is:

constructor Create(pSizeX, pSizeY, pDepth: integer; c: T = 0);

You can access the data as 1D or 3D with:

property Raw[x: integer]: T read GetRaw write SetRaw;
property Data[x, y, d: integer]: T read Get write Store; default;

Your code will look like this:

// Usage Examples
vInput := TNNetVolume.Create(32, 32, 3);
vInput[1, 1, 1] := 1;
vInput[2, 2, 2] := vInput[1, 1, 1] + 1;
vInput.Raw[10] := 5;

vInput.RandomizeGaussian();
WriteLn('Avg: ', vInput.GetAvg());
WriteLn('Variance: ', vInput.GetVariance());
WriteLn('Std Dev: ', vInput.GetStdDeviation());

WriteLn('Multiplying by 10');
vInput.Mul(10);
WriteLn('Avg: ', vInput.GetAvg());
WriteLn('Variance: ', vInput.GetVariance());
WriteLn('Std Dev: ', vInput.GetStdDeviation());

As examples, you can add, subtract, multiply and calculate dot products with:

procedure Add(Original: TNNetVolume); overload;
procedure Sub(Original: TNNetVolume); overload;
procedure Mul(Value: Single); overload;
function DotProduct(Original: TNNetVolume): TNeuralFloat; overload;

In the case that you need the raw position or raw pointer to an element of the volume, you can get with:

function GetRawPos(x, y, d: integer): integer; overload;
function GetRawPos(x, y: integer): integer; overload;
function GetRawPtr(x, y, d: integer): pointer; overload;
function GetRawPtr(x, y: integer): pointer; overload;
function GetRawPtr(x: integer): pointer; overload;

You can easily operate volumes with OpenCL via TEasyOpenCLV:

  TEasyOpenCLV = class (TEasyOpenCL)
    public
      function CreateBuffer(flags: cl_mem_flags; V: TNNetVolume): cl_mem; overload;
      function CreateInputBuffer(V: TNNetVolume): cl_mem; overload;
      function CreateHostInputBuffer(V: TNNetVolume): cl_mem; overload;
      function CreateOutputBuffer(V: TNNetVolume): cl_mem; overload;
      function CreateBuffer(V: TNNetVolume): cl_mem;  overload;

      function WriteBuffer(buffer: cl_mem; V: TNNetVolume; blocking: cl_bool = CL_FALSE): integer;
      function ReadBuffer(buffer: cl_mem; V: TNNetVolume; blocking: cl_bool = CL_TRUE): integer;

      function CreateAndWriteBuffer(V: TNNetVolume; var buffer: cl_mem): integer; overload;
      function CreateAndWriteBuffer(V: TNNetVolume): cl_mem; overload;
      function CreateWriteSetArgument(V: TNNetVolume; kernel:cl_kernel; arg_index: cl_uint): cl_mem;
      function CreateOutputSetArgument(V: TNNetVolume; kernel:cl_kernel; arg_index: cl_uint): cl_mem;
  end;

Volume Pairs, Volume Lists and Volume Pair Lists

Volumes can be organized in pairs:

  /// Implements a pair of volumes
  TNNetVolumePair = class(TObject)
    protected
      FA: TNNetVolume;
      FB: TNNetVolume;
    public
      constructor Create(); overload;
      constructor Create(pA, pB: TNNetVolume); overload;
      constructor CreateCopying(pA, pB: TNNetVolume); overload;

      destructor Destroy(); override;

      property A:TNNetVolume read FA;
      property B:TNNetVolume read FB;
      property I:TNNetVolume read FA;
      property O:TNNetVolume read FB;
  end;

Depending on the problem that you are trying to solve, modelling the training with pairs or pair lists might be helpful. Typically, a pair will be (input, desired output). This is how volume lists and volume pair lists have been implemented:

TNNetVolumeList = class (specialize TFPGObjectList<TNNetVolume>
TNNetVolumePairList = class (specialize TFPGObjectList<TNNetVolumePair>)

Neural Network Layers

The layered structure of artificial neural networks is inspired by the organization of the human brain and nervous system. In the human brain, information processing occurs in a hierarchical manner. Sensory inputs are first processed by lower-level neurons, which extract simple features. These features are then passed on to deeper neurons that combine them to recognize more complex patterns. This hierarchical processing is mirrored in artificial neural networks through the use of stacked layers.

Biological neurons are connected to each other through synapses, forming complex networks. Similarly, in artificial neural networks, neurons in one layer are connected to neurons in the next layer, mimicking this interconnected structure. Biological neurons fire (activate) based on a non-linear response to their inputs. This non-linearity is crucial for the brain's ability to learn complex patterns. In artificial neural networks, we use non-linear activation functions (such as ReLU) to introduce this non-linearity. Different regions of the brain specialize in processing different types of information. For instance, the visual cortex has layers specialized for detecting edges, shapes, and complex objects. This specialization is reflected in artificial neural networks, where different layers can learn to recognize different levels of abstraction.

In the context of artificial neural networks, we can see this biologically-inspired layered approach implemented. For example:

NN := TNNet.Create();
NN.AddLayer([
  TNNetInput.Create(32, 32, 3),
  TNNetConvolutionLinear.Create({neurons=}16, {featuresize}3, {padding}1, {stride}1),
  TNNetReLU6.Create()
]);

This code snippet demonstrates the creation of a neural network with an input layer and a convolutional layer followed by a ReLU6 activation. This structure is inspired by the visual cortex in the brain, where neurons respond to specific patterns in their receptive fields, similar to how convolutional layers operate. The CAI Neural API also supports the creation of more complex, biologically-inspired architectures. These architectures are designed with multiple layers of different types, mirroring the complex structure of the brain.

Artificial neural networks with multiple layers and specialized structures are inspired by the hierarchical and specialized nature of biological neural processing. It's important to note that while artificial neural networks are inspired by biological neural networks, they are highly simplified models. The human brain is far more complex, with various types of neurons, complex connectivity patterns, and mechanisms we don't yet fully understand. However, the layered structure in artificial neural networks has proven to be a powerful approach for solving complex problems in machine learning, inspired by the remarkable capabilities of biological neural networks.

Input Layer

The input layer serves as the gateway to the entire network. It's like the sensory organs of our brain, receiving information from the outside world. Without an input layer, the neural network would have no way to receive and interpret the initial data, making it impossible to perform any meaningful computations or learning tasks. The TNNetInput class implements the input layer.

Fully Connected (Dense) Layers

Fully connected layers, also known as dense layers, are a fundamental component of neural networks. In these layers, every neuron is connected to every neuron in the previous layer, allowing for comprehensive information processing across the entire network.

In the context of the CAI Neural API, fully connected layers are represented by various classes derived from TNNetFullConnect. These layers play a crucial role in transforming input data and learning complex patterns.

The computation process in a fully connected layer involves:

1. Multiplying input values by the layer's weights.
2. Adding bias terms (if not suppressed).
3. Applying an activation function (if present).

Key types of fully connected layers include:

1. TNNetFullConnectLinear: a basic fully connected layer without an activation function. It performs a linear transformation of the input data.
2. TNNetFullConnectReLU: incorporates the Rectified Linear Unit (ReLU) activation function. ReLU introduces non-linearity by outputting the input for positive values and zero for negative values, helping the network learn complex patterns.
3. TNNetFullConnectSigmoid: applies the sigmoid activation function to the layer's output. Sigmoid squashes the output between 0 and 1, useful for binary classification tasks.
4. TNNetComplexLinear: the 2-dimensional complex base rung of the same Cayley–Dickson family — each 2-channel group (Re,Im) is multiplied by a learned complex w = a + b·i via the 2×2 complex-multiply block [[a,-b],[b,a]] (Re' = a·Re − b·Im, Im' = a·Im + b·Re), using ~1/2 the weights of an equal-width real dense layer while still mixing the real and imaginary parts. The forward block equals the true complex product and is norm-multiplicative |w·X| = |w|·|X|. Input/output Depth must be multiples of 2. See examples/ComplexLinear/.
5. TNNetQuaternionLinear: a hypercomplex dense layer that shares each 4×4 Hamilton block from a single learned quaternion, coupling 4-channel groups (rotation/scaling in quaternion space) with ~1/4 the weights of an equal-width real dense layer. Input/output Depth must be multiples of 4.
6. TNNetOctonionLinear: the 8-dimensional octonion (Cayley–Dickson) generalization of the above — each 8-channel group is multiplied by a learned octonion via the (non-associative) octonion product, assembled as an 8×8 signed block, using ~1/8 the weights of an equal-width real dense layer. The implementation hard-codes an auditable Cayley–Dickson sign table verified by a norm-multiplicativity test (|W·X| = |W|·|X|). Input/output Depth must be multiples of 8. See examples/OctonionLinear/.

Fully connected layers are typically used in neural network architectures as:

1. Hidden layers for processing and transforming features.
2. Output layers for producing final predictions.

For example, in the provided context, we see a simple neural network structure:

NN := TNNet.Create();
NN.AddLayer([
  TNNetInput.Create(2),       // Input layer with 2 inputs
  TNNetFullConnect.Create(2), // Hidden fully connected layer with 2 neurons
  TNNetFullConnect.Create(1)  // Output fully connected layer with 1 neuron  
]);

Convolutional Layers

Neurons, filters, and kernels are often used as synonyms in the context of neural networks, particularly in convolutional neural networks (CNNs). They are closely related concepts that are used interchangeably. Here's why:

• Neurons: in artificial neural networks, neurons are the basic computational units. They receive input, process it, and produce an output. In the context of CNNs, the term "neuron" is sometimes used to refer to a single element in a feature map.
• Filters: in CNNs, filters (also called convolution kernels) are small matrices of weights that slide over the input data to detect specific features. Each filter produces a feature map in the output layer.
• Kernels: in image processing and CNNs, kernels are small matrices used for various operations like blurring, sharpening, or edge detection. In the context of CNNs, kernels and filters are essentially the same thing. The reason these terms are often used synonymously is that they all contribute to the feature detection and transformation process in neural networks:
  • A single filter/kernel can be thought of as a specialized neuron that detects a specific feature across the entire input.
  • The weights in a filter/kernel are analogous to the weights in a traditional neuron.
  • The output of applying a filter/kernel to an input region is similar to the activation of a neuron in response to its inputs.

In practice, when implementing CNNs, the terms "filter" and "kernel" are more commonly used than "neuron" when referring to the convolutional layers. However, the underlying concept of a computational unit that processes input and produces output remains the same across these terms.

Convolutional layers are fundamental building blocks in neural networks, particularly in the field of computer vision and image processing. They are designed to automatically and adaptively learn spatial hierarchies of features from input data, such as images.

In the context of the CAI Neural API, convolutional layers are implemented as classes derived from TNNetConvolutionAbstract. This abstract base class provides the core functionality for convolutional operations.

The structure of a convolutional layer typically includes:

1. Input: A multi-dimensional array (usually 3D for images: width, height, and channels).
2. Kernels (or filters): small matrices of weights that slide over the input.
3. Feature maps: the output produced by applying the kernels to the input.

Key parameters of convolutional layers include:

• Number of features (or filters).
• Feature size (kernel size).
• Padding.
• Stride.

The CAI Neural API offers several types of convolutional layers:

1. TNNetConvolution: the standard convolutional layer.
2. TNNetConvolutionLinear: a convolutional layer without an activation function.
3. TNNetConvolutionReLU: a convolutional layer with a ReLU activation function.
4. TNNetConvolutionRectangular / TNNetConvolutionRectangularReLU: rectangular (asymmetric) kernel convolutions taking independent FeatureSizeX and FeatureSizeY, for the 1xN / Nx1 spatially-factorized convolutions used in Inception-v3 (1x7 + 7x1 branches), SegFormer/Mix-FFN, and large-receptive-field stems.
5. TNNetComplexConv: the 2-dimensional complex base rung of the same Cayley–Dickson convolution family — each 2-channel input patch (Re,Im) is complex-multiplied by a learned complex filter tap w = a + b·i (Re' = a·Re − b·Im, Im' = a·Im + b·Re) and accumulated, coupling the real and imaginary parts across space using only ~1/2 the weights of an equal-width real convolution. Input Depth and the feature count must both be multiples of 2. It reuses the gradient-checked 2×2 complex-multiply forward/backward kernel of its dense sibling TNNetComplexLinear. See examples/ComplexLinear/.
6. TNNetQuaternionConv: a hypercomplex convolution whose per-output-channel filter taps are quaternion weights — each 4-channel input patch is Hamilton-multiplied by a learned quaternion and accumulated, so the four channel components are coupled (rotation/scaling in quaternion space) using only ~1/4 the weights of an equal-width real convolution. Input Depth and the feature count must both be multiples of 4. It reuses the gradient-checked 4×4 Hamilton forward/backward kernel of its dense sibling TNNetQuaternionLinear (the library's first hypercomplex layer). See examples/QuaternionConv/ and examples/QuaternionLinear/.
7. TNNetOctonionConv: the 8-dimensional octonion (Cayley–Dickson) generalization of TNNetQuaternionConv — each 8-channel input patch is octonion-multiplied by a learned octonion filter tap and accumulated, coupling all eight channel components across space using only ~1/8 the weights of an equal-width real convolution. Input Depth and the feature count must both be multiples of 8. It reuses the same auditable, norm-multiplicativity-verified 8×8 Cayley–Dickson forward/backward kernel as its dense sibling TNNetOctonionLinear. See examples/OctonionConv/ and examples/OctonionLinear/.

Convolutional layers are crucial in neural networks because they:

1. Automatically learn hierarchical features from data.
2. Maintain spatial relationships in the input.
3. Reduce the number of parameters compared to fully connected layers.
4. Enable the network to be translation-invariant.

In practice, convolutional layers are often used in combination with other layer types, such as pooling layers (e.g., TNNetMaxPool) and normalization layers (e.g., TNNetMovingStdNormalization), to create powerful neural network architectures for tasks like image classification, object detection, and segmentation.

Here's a brief example of how to create a convolutional layer using the CAI Neural API:

NN := TNNet.Create();
NN.AddLayer([
  TNNetInput.Create(32, 32, 3),  // Input layer for 32x32 RGB images
  TNNetConvolutionLinear.Create(
    {Features=}64,     // Number of output features
    {FeatureSize=}5,   // 5x5 kernel size
    {Padding=}2,       // Padding of 2 pixels
    {Stride=}1,        // Stride of 1 pixel
    {SuppressBias=}1   // Suppress bias
  ),
  TNNetReLU6.Create()  // Activation function
]);

This example creates a convolutional layer with 64 features, a 5x5 kernel size, padding of 2, and a stride of 1, followed by a ReLU6 activation function.

These are tha available convolutional layers in CAI:

Grouped pointwise convolutions are an interesting and efficient variant of standard convolutions in neural networks. Grouped pointwise convolutions are a type of convolution operation where the input channels are divided into groups, and each group is processed separately. This is particularly useful for 1x1 convolutions (pointwise) where the spatial dimensions are not affected. The grouped approach can significantly reduce the number of parameters in a neural network as shown in the papers Grouped Pointwise Convolutions Reduce Parameters in Convolutional Neural Networks and An Enhanced Scheme for Reducing the Complexity of Pointwise Convolutions in CNNs for Image Classification Based on Interleaved Grouped Filters without Divisibility Constraints. By reducing parameters, these convolutions can make models more efficient in terms of computation and memory usage. These convolutions can be combined with other techniques like normalization and intergroup connections. This flexibility allows for the creation of more sophisticated network designs. Grouped pointwise convolutions are particularly useful in efficient network designs, such as mobile or edge computing applications where resource constraints are significant. They allow for maintaining model expressivity while reducing computational requirements.

The grouped pointwise convolutional layers are:

Locally Connected Layers

A locally connected layer is a type of neural network layer that shares some similarities with convolutional layers but has some distinct characteristics:

• Structure: Locally connected layers, like convolutional layers, operate on local regions of the input. However, unlike convolutional layers, they do not share weights across different positions in the input.
• Weight independence: Each local region in the input has its own set of weights, which are not shared with other regions. This allows the layer to learn position-specific features.
• Flexibility: Locally connected layers offer more flexibility in learning spatial hierarchies compared to fully connected layers, while still maintaining position-specific information unlike convolutional layers.
• Parameters: These layers typically have more parameters than convolutional layers due to the lack of weight sharing, which can lead to increased computational complexity and memory usage.
• Use cases: Locally connected layers can be useful in scenarios where position-specific features are important, such as in face recognition tasks where different parts of the face have distinct characteristics based on their location.

Min / Max / Avg Pools

Max, min, and avg poolings are downsampling techniques used in neural networks, particularly in convolutional neural networks (CNNs). Let's explore each of these pooling types as implemented in the CAI Neural API:

1. Max Pooling (TNNetMaxPool): Max pooling selects the maximum value from a defined region of the input.
   • It reduces spatial dimensions while retaining the most prominent features.
   • Useful for detecting specific features regardless of their position in the input.
2. Min Pooling (TNNetMinPool): Min pooling selects the minimum value from a defined region of the input.
   • It can be useful for detecting dark features or gaps in the input.
   • Less common than max pooling but valuable in specific scenarios.
3. Average Pooling (TNNetAvgPool): Average pooling calculates the average value of a defined region of the input.
   • It smooths the input and can help in reducing noise.
   • Often used when we want to preserve more contextual information compared to max pooling.

Unique pooling variants in the API:

• TNNetMinMaxPool: Performs both max and min pooling and concatenates the results.
• TNNetAvgMaxPool: Combines average and max pooling.
• TNNetLpPool: generalized Lp pooling y = ((1/N)·Σ|xᵢ|^p)^(1/p) over each window, with a configurable real exponent p (TNNetLpPool.Create(PoolSize, Stride, Padding, p), default p=2). p=1 is mean-of-absolute-values, p=2 is RMS pooling, and large p approaches max pooling — a single knob interpolating between average and max pooling. Its analytic backward pass ∂y/∂xᵢ = (y^(1-p)/N)·|xᵢ|^(p-1)·sign(xᵢ) is numerically gradient-checked.
• TNNetSoftPool: exponentially-weighted ("softmax") pooling (Stergiou, Poppe & Kalliatakis, 2021). Over each window it computes wᵢ = exp(β·xᵢ)/Σⱼexp(β·xⱼ) and y = Σᵢ wᵢ·xᵢ (TNNetSoftPool.Create(PoolSize, Stride, Padding, β), default β=1; window softmax stabilised by subtracting the window max). The optional inverse-temperature β is a single knob spanning the average↔max family: β → ∞ recovers max pooling, β → 0 recovers average pooling, and β = 1 is the original SoftPool. Unlike max pooling every cell receives gradient: its analytic backward pass ∂y/∂xᵢ = wᵢ·(1 + β·(xᵢ − y)) is numerically gradient-checked across a β sweep.
• TNNetStochasticPool: stochastic pooling (Zeiler & Fergus, 2013). Over each window it builds a probability distribution pᵢ = aᵢ/Σⱼaⱼ from the (assumed non-negative, e.g. post-ReLU) activations. While training (toggled on by TNNet.EnableDropouts(true), like dropout) it samples one cell with probability pᵢ and outputs it, routing the whole window gradient to that sampled cell (like max pooling routes to its argmax); at inference it is deterministic, outputting the probability-weighted expectation y = Σᵢ pᵢ·aᵢ. Sampling uses the library RNG so it is reproducible under a fixed RandSeed. If a window sum is ≤ 0 (degenerate / negative activations) it falls back to the plain window mean. Constructor params (size/stride/padding) match TNNetMaxPool; assumes square feature maps (SizeX = SizeY).

Backpropagation in pooling layers: During backpropagation, pooling layers distribute the gradient differently:

• Max Pooling: The gradient is passed only to the neuron that had the maximum value during the forward pass.
• Min Pooling: Similar to max pooling, but for the minimum value.
• Average Pooling: The gradient is divided equally among all neurons in the pooling region.

The CAI Neural API implements these backpropagation methods in the respective Backpropagate() functions of each pooling class.

Deconvolution (Upsampling) counterparts: The API also provides deconvolution or upsampling layers, which can be seen as the inverse operations of pooling:

• TNNetDeMaxPool: a deconvolution layer that can upsample the input.
• TNNetUpsample: also known as depth_to_space, this layer can increase the spatial dimensions of the input.

These layers are crucial in architectures like autoencoders or in tasks requiring upsampling, such as image segmentation.

When to use each pooling type:

• Max Pooling: it is useful for detecting features regardless of their exact location. It's commonly used in classification tasks.
• Min Pooling: it is useful when the absence of features is important, or when working with inverted data.
• Average Pooling: it is good for preserving more context and reducing noise. Often used in later layers of the network.
• TNNetMinMaxPool: used when you want to capture both the presence and absence of features.
• TNNetAvgMaxPool: used when you need to balance between preserving prominent features and maintaining context.

The CAI Neural API also provides specialized versions:

• TNNetMaxChannel and TNNetMinChannel: perform max and min operations across the entire channel into a single number per channel.
• TNNetAvgChannel: averages the entire channel into a single number per channel.

Trainable Normalization Layers Allowing Faster Learning/Convergence

Normalization layers may offer:

• Improved training stability.
• Better generalization.
• Potential for faster convergence.

The available normalization techniques are:

• Zero-centering (TNNetChannelZeroCenter).
• Standard deviation normalization (TNNetMovingStdNormalization, TNNetChannelStdNormalization).
• Per-sample layer normalization (TNNetLayerNorm, TNNetRMSNorm, TNNetGroupNorm).

See the Normalization cheat sheet for a side-by-side comparison of every normalization layer (axes reduced over, learnable parameters, formula, and when to use each).

TNNetLayerNorm normalizes each input sample over all its elements (SizeX*SizeY*Depth) to zero mean and unit variance, then applies a learnable per-element scale (gamma) and bias (beta). Unlike batch normalization it does not depend on batch statistics, which makes it well suited to transformers and recurrent models. Add it with NN.AddLayer(TNNetLayerNorm.Create());.

TNNetRMSNorm is a cheaper, transformer-friendly variant that divides each sample by the root mean square of its elements (no mean subtraction) and applies a learnable per-element scale. Add it with NN.AddLayer(TNNetRMSNorm.Create());.

TNNetRMSNormGated keeps the same RMS normalization but replaces the per-element scale with a learnable per-channel sigmoid gate sigmoid(g[d]). The gate logits are initialised to 0, so an untrained layer halves each normalized activation (sigmoid(0) = 0.5) and the channels open or close independently during training. Add it with NN.AddLayer(TNNetRMSNormGated.Create());.

TNNetSwitchableNorm lets the network learn how much LayerNorm vs RMSNorm to apply to the same input. It computes both a LayerNorm-style normalization L = (x - mean)/sqrt(var + eps) and an RMSNorm-style normalization R = x / sqrt(mean(x^2) + eps) per sample, then blends them with a softmax over two learnable scalar logits: y = a_ln * L + a_rms * R with (a_ln, a_rms) = softmax(w_ln, w_rms). There is no per-element gamma/beta; the only parameters are the two mixing logits, both initialised to 0 so an untrained layer is an exact 50/50 blend. Add it with NN.AddLayer(TNNetSwitchableNorm.Create());.

TNNetGroupNorm splits the input channels (Depth) into Groups contiguous groups and normalizes each group independently, then applies a learnable per-element scale and bias. Depth must be divisible by Groups; otherwise it falls back to a single group. Pass the group count to the constructor, e.g. NN.AddLayer(TNNetGroupNorm.Create(8));.

Non Trainable and per Sample Normalization Layers

Normalization layers (TNNetLayerMaxNormalization, TNNetLayerStdNormalization, TNNetLocalResponseNorm2D, TNNetLocalResponseNormDepth) help stabilize training and can improve model performance by managing the scale and distribution of activations. They are particularly useful in deep networks where the scale of values can change dramatically between layers.

TNNetLayerMaxNormalization normalizes based on the maximum value, while TNNetLayerStdNormalization uses standard deviation. These are particularly useful when you want to normalize the activations within a specific range or distribution without learning any parameters. They can be applied to various network architectures and are especially helpful when dealing with varying scales of input features.

TNNetLocalResponseNorm2D and TNNetLocalResponseNormDepth implement types of local Response Normalization (LRN). LRN is inspired by lateral inhibition in real neurons. It's particularly useful in Convolutional Neural Networks (CNNs) for image processing tasks. You may use it in scenarios where you want to create competition amongst neuron outputs in the same layer.

TNNetLocalResponseNorm2D is applied across nearby kernel maps at the same spatial position, while TNNetLocalResponseNormDepth normalizes across the depth dimension. These layers can help in increasing the generalization capability of the model, reducing the chances of overfitting and enhancing the model's ability to detect high-frequency features with a big response.

Random layers (TNNetRandomMulAdd, TNNetChannelRandomMulAdd) serve as powerful regularization techniques, helping to prevent overfitting and improve the model's ability to generalize. They can be especially beneficial when working with limited datasets or when you want your model to be robust to small variations in input.

These layers provide various tools for normalization, regularization, and introducing controlled variability in neural networks. The choice of which layers to use and where to place them in your network architecture depends on the specific problem you're trying to solve, the characteristics of your data, and the behavior you want to encourage in your model.

Concatenation, Summation and Reshaping Layers

These layers are essential for creating flexible and powerful neural network architectures. Let's break them down:

1. Concatenation Layers: There are two main types of concatenation layers in the CAI Neural API: a. TNNetConcat:
   • This layer concatenates outputs from multiple layers along the depth dimension.
   • It's designed to work with layers that have the same spatial dimensions (X and Y sizes).
   • Usage: It's particularly useful when you want to combine features from different processing paths in your network. b. TNNetDeepConcat:
   • This layer also concatenates outputs from multiple layers, but it's specifically optimized for the depth dimension.
   • It maintains separate arrays to track the depths of each layer and channel, allowing for efficient deep concatenation.
   • Usage: Ideal for creating architectures that process information in parallel and then combine the results.
2. Summation Layer (TNNetSum):
   • This layer adds together the outputs of multiple layers element-wise.
   • It's designed to work with layers of the same size.
   • Usage: Commonly used in residual network (ResNet) style architectures, where it allows for skip connections that help mitigate the vanishing gradient problem and enable the training of very deep networks.

These layers provide several benefits in neural network design:

1. Flexibility: they allow for the creation of complex, non-linear network topologies that can process information in parallel and then combine it in various ways.
2. Feature Fusion: concatenation and summation layers enable the network to combine features from different processing streams, potentially capturing multi-scale or multi-aspect information.
3. Skip Connections: summation layers are crucial for implementing skip connections, which are fundamental to many modern architectures like ResNets and DenseNets.
4. Dimensionality Manipulation: the transposition layers allow for creative manipulations of data dimensions, which can be crucial for certain types of operations or for interfacing between different parts of a network.
5. Custom Architectures: these layers provide the building blocks for designing novel network architectures tailored to specific tasks or data types.

By using these layers creatively, developers can build highly customized and efficient neural network architectures that are optimized for their specific use cases.

Embedding heads

For contrastive / metric-learning models the goal is not to classify but to embed: map each input to a vector so that semantically-similar inputs land close together and dissimilar ones far apart. Three layers compose into such a head:

How to build a contrastive / metric-learning head. Build a small embedding sub-net (an MLP or conv trunk) that maps the input to an embed_dim vector, end it with TNNetL2Normalize so embeddings live on the unit sphere, then train it with TNNetTripletLoss. Because the triplet head takes no external target — supervision is implicit in its anchor|positive|negative depth layout — you feed it three embeddings at once. The cleanest fully-native way is a weight-shared siamese net: feed the triplet as three spatial positions, embed each with pointwise (featuresize=1) layers so the same weights apply to all three, TNNetL2Normalize, then TNNetReshape(1, 1, 3*embed_dim) to lay the three embeddings out as the a|p|n depth chunks the loss head consumes. At inference, drop the loss head and read the embedding directly; use TNNetCosineSimilarity (or a plain dot product on unit-norm vectors) to score pairs. See the worked Triplet embedding example.

Split Channels

TNNetSplitChannels and TNNetSplitChannelEvery are specialized layer types in the CAI Neural API that allow for selective channel manipulation within neural networks.

1. TNNetSplitChannels: This layer is designed to pick or split selected channels from the previous layer. It provides fine-grained control over which specific channels are passed on to subsequent layers in the network. Key features:

   • It can be created with a specific range of channels (ChannelStart and ChannelLen) or with an array of specific channel indices.

   Potential uses:

   • Feature selection: Allowing the network to focus on specific features represented by certain channels.
   • Creating multiple parallel paths in the network that process different subsets of the input channels.
   • Implementing attention-like mechanisms by selectively passing certain channels forward.

2. TNNetSplitChannelEvery: This layer is a specialized version of TNNetSplitChannels. It splits channels at regular intervals.

   Potential uses:

   • Creating regular patterns of channel selection throughout the network.
   • Implementing a form of grouped convolutions or channel-wise operations.
   • Reducing the computational load by consistently selecting a subset of channels at regular intervals.

Both these layers offer powerful tools for manipulating the flow of information through the network's channels. They allow for the creation of more complex and efficient network architectures by providing fine control over which features (represented by channels) are processed in different parts of the network.

These layers could be particularly useful in scenarios where:

• You want to reduce the computational complexity of your model by focusing on the most important channels.
• You're designing a network with multiple parallel paths, each operating on different subsets of the input features.
• You're implementing custom attention mechanisms or feature selection techniques within your network.

Picking the right channel-select layer. Three closely related layers select depth channels — pick by the shape of the selection:

• TNNetGather(Channel) selects a single channel (Output[x,y,0] := Input[x,y,Channel], output depth 1) — the degenerate one-index case.
• TNNetSplitChannels selects a contiguous range (Create(ChannelStart, ChannelLen)) or an explicit list, and is the right tool for plain slicing / parallel-path splits.
• TNNetGatherChannels([i0, i1, ...]) selects an arbitrary, ordered, possibly-repeated index list (Output[x,y,k] := Input[x,y,Channels[k]], output depth = list length), so it doubles as a learnable-free channel reorder / prune / duplicate. Add it via the convenience builder TNNet.AddGatherChannels([...]). See the runnable examples/GatherChannelsRouting/ demo. Repeats are allowed; backward accumulates the duplicated output errors onto the shared source channel.
• TNNetGatherTokens([i0, i1, ...]) is the X-axis (token/sequence) analogue of TNNetGatherChannels: over a (SizeX, 1, Depth) sequence it selects an ordered subset of positions (Output[k,0,d] := Input[Tokens[k],0,d], output length = list length). It is the token-reordering primitive behind the Swin Transformer window-partition / window-reverse and cyclic-shift (a fixed permutation of the (H*W,1,C) token grid into window-contiguous order, and its inverse). Backward scatters each output position's error back to its source with Add (repeats accumulate).

Transposing Layers

The layers TNNetTransposeXD and TNNetTransposeYD are specialized layer types in the CAI Neural API that perform specific transposition operations on the input data. These transposition operations can be particularly useful in various neural network architectures and data processing pipelines:

• Reshaping Data: they allow for flexible reshaping of data between different network layers, which can be crucial for certain model designs.
• Feature Manipulation: by swapping spatial and depth dimensions, these layers can help in reorganizing feature representations, which might be beneficial for subsequent processing steps.
• Dimension Reduction or Expansion: depending on the input shape, these transpositions can effectively reduce or expand certain dimensions, potentially helping in compressing or expanding feature representations.
• Adapting to Different Input Formats: these layers can be useful when dealing with data that comes in different formats or when interfacing between different parts of a neural network that expect data in specific shapes.
• Custom Architecture Designs: they provide flexibility in designing custom neural network architectures that may require unconventional data flows between layers.

These layers are implemented with both forward (Compute) and backward (Backpropagate) methods, indicating that they are fully integrated into the network's training process and can be used in the middle of a network, not just as preprocessing steps. This can be particularly valuable for researchers and practitioners working on novel network designs or dealing with unconventional data structures.

Layers with Activation Functions and no Trainable Parameter

Activation functions are a fundamental component of neural networks. These functions play several crucial roles in neural networks:

• Introducing non-linearity: this allows the network to model complex, non-linear relationships in data.
• Normalizing outputs: many activation functions map inputs to a fixed range, helping to prevent issues like exploding gradients.
• Representing features: different activation functions can help in capturing various types of patterns or features in the data. The choice of activation function can significantly impact the performance and learning capabilities of a neural network, and different problems may benefit from different activation functions.

The CAI Neural API supports various types of activation functions, as per the below table:

Gated Linear Units

Gated Linear Units split the input along the channel (depth) axis into two equal halves A and B, and output A multiplied by a gating activation applied to B. The output depth is therefore half of the input depth, and the input depth must be even. These layers have no trainable parameters. They are commonly used inside transformer feed-forward blocks (https://arxiv.org/abs/2002.05202))

[...]