README

Dolphin 🐬

Dolphin is a Delphi-native Pascal GPT framework for building and iterating on GPT-style models.
It targets developers who want full control over tokenization, vocabulary, and neural network logic — implemented in Object Pascal.

Logserver - this UI can be found in the logserver folder

Features

Core Architecture

• GPT-like architecture implemented entirely in Delphi
• Modular feed-forward and transformer blocks for rapid iteration and benchmarking
• Pre-Norm architecture with residual connections
• Multiple activation functions — GELU, ReLU, Swish, SELU, ELU, LeakyReLU

GPU Acceleration (CUDA)

• NVIDIA CUDA support — Automatic CPU/GPU dispatch based on computational intensity
• Hybrid precision — FP64 on CPU (Delphi Double / MrMath default) and FP32 on GPU for performance
• GPU Memory Pool — Intelligent buffer reuse and memory management
• Weight caching — Persistent GPU-resident weights with cache hit tracking
• Native CUDA kernels — MatMul, Softmax, GELU, LayerNorm, RoPE, AdamW, and more
• See CUDA Documentation for detailed kernel reference

Training Features

• Training Objectives — Control pipeline stop points or modes (TokenizerOnly, EmbeddingsOnly, SentenceEmbeddings, PretrainOnly, FullModel, FineTune)
• True Batch Training — Efficient batch processing with gradient accumulation
• Micro Batching — Memory-efficient training with configurable batch subdivision
• GPU Auto Batch Size — Automatic batch size optimization based on available VRAM
• LLM-driven Hyperparameter Tuning (Optional) — At epoch end, an external LLM can propose hyperparameter updates (for example LearningRate, GradientClipNorm, Dropout, WeightDecay, and micro-batch settings). Requires LLMTUNNING and optional MakerAi integration.
• Sequence Modes:
  • Packed (Stream) — Concatenates sequences for maximum throughput
  • Chunk — Splits long sequences with configurable stride for overlap
  • Truncate — Clips sequences to maximum length
• Early Stopping — Monitors validation loss/perplexity with patience and best-weights restoration
• JSONL TrainingLogs — Real-time training progress with Python console support

Data Science & DataFrames

• TDataFrame Component — A columnar, Pandas-like DataFrame for data science and ML data loading
• Native Data Ingestion — Load directly from any TDataSet (e.g., FireDAC TFDMemTable, SQL queries) with automatic type mapping and pre-allocation, alongside multi-threaded CSV loading.
• Structured Data Processing — Min/Max scaling, Z-Score Normalization, and Label Encoding.
• Data Shuffling — Fast Fisher-Yates row shuffling via virtual indices (zero data copying).
• GPU Integration & Batching — Direct O(1) batch extraction to TDeviceMatrix without intermediate memory copies.
• IDataProvider Architecture — Generic TDataFrameProvider wraps DataFrames to yield TTensorBatch (separate Features and Targets matrices) for training loops, cleanly decoupling structured data from NLP token pipelines.
• Zero-Copy Dataset Splits — Dynamically split DataFrames into Train and Validation views using index slicing without duplicating data in memory.
• LoRA & Fine-Tuning Ready — Feeds purely numeric feature/target tensors directly into the network, bypassing token embeddings for structured tabular training.

Positional Encoding

• Sinusoidal Positional Encoding — Classic fixed positional embeddings
• Rotary Position Embeddings (RoPE) — Modern relative position encoding with GPU acceleration

Tokenization & Vocabulary

• Custom BPE tokenizer (Byte Pair Encoding) with vocabulary management
• SQLite database backend for persistent vocabulary and merge rules
• Hugging Face export — Vocabulary/merges export compatible with Hugging Face tokenizers

Precision & Memory

• FP16 Matrix Support — Half-precision floating point for reduced memory usage
• Optimized memory management — Pooled allocations, buffer reuse

Generation & Inference

• Advanced Generation — Top-K, Top-P (Nucleus), and Temperature sampling
• Greedy decoding — Deterministic output

Configuration

• Flexible Configuration — JSON-based configuration and CLI arguments
• Offline and local-friendly — No external runtimes required
• Multi-GPU Support — Specify exactly which GPU the model will use using configuration or query available GPUs at runtime.

Upcoming features

• LoRA (Low-Rank Adaptation) support for parameter-efficient fine-tuning (train small adapter matrices while keeping base weights frozen). For more information: https://huggingface.co/docs/diffusers/training/lora

Available Units

Dependencies

• Required: MrMath, a high-performance numerical library for Delphi.
  It provides matrix operations, linear algebra, and numerical routines similar in purpose to Python's NumPy, serving as the computational backbone for neural and tokenization modules.
• Optional (only when using LLM Optimizer): MakerAi.
• LLM Optimizer usage note:
  • Enable LLMTUNNING only if you want LLM-driven hyperparameter proposals.
  • If you do not want the optimizer, leave LLMTUNNING undefined and keep UseLLMOptimizer disabled.

Architecture Overview

           ?????????????????????????????????????????????????????????????????????????????????????????????
           ???         Input Text          ???
           ?????????????????????????????????????????????????????????????????????????????????????????????
                          ???
                          ???
               ??????????????????????????????????????????????????????????????????
               ???  BPE Tokenizer     ???
               ??? (PascalGPT.BPETokenizer) ???
               ??????????????????????????????????????????????????????????????????
                          ???
                   Tokens (IDs)
                          ???
                          ???
             ????????????????????????????????????????????????????????????????????????????????????
             ???   Embedding Layer        ???
             ??? (PascalGPT.Embeddings)   ???
             ???  + Sinusoidal / RoPE     ???
             ????????????????????????????????????????????????????????????????????????????????????
                          ???
                          ???
          ??????????????????????????????????????????????????????????????????????????????????????????????????????
          ???       Transformer Block         ???
          ??? (PascalGPT.TransformerBlock)    ???
          ???  ?????? Attention (multi-head)      ???
          ???  ???   ?????? RoPE / Causal Mask      ???
          ???  ?????? FeedForward (GELU/ReLU)     ???
          ??????????????????????????????????????????????????????????????????????????????????????????????????????
                          ???
                          ??? (N layers)
                          ???
                          ???
             ????????????????????????????????????????????????????????????????????????????????????
             ???   Output Layer / Logits  ???
             ???(Prediction of next token)???
             ????????????????????????????????????????????????????????????????????????????????????
                          ???
                          ???
           ?????????????????????????????????????????????????????????????????????????????????????????????
           ???     Decoded Text Output     ???
           ?????????????????????????????????????????????????????????????????????????????????????????????

Training Configuration

Logserver - this UI can be found in the logserver folder

Sequence Modes

Early Stopping

{
  "EarlyStopping": {
    "Enabled": true,
    "Monitor": "val_loss",
    "Mode": "min",
    "MinDelta": 0.01,
    "Patience": 3,
    "MinEpochs": 5,
    "RestoreBestWeights": true
  }
}

Micro Batching

Enables training with large effective batch sizes on limited VRAM:

{
  "Training": {
    "MicroBatchSize": 4,
    "MicroBatchMaxTokens": 2048
  }
}

Training Objectives

Control where the training pipeline stops, enabling specialized workflows for different use cases:

Configuration:

{
  "General": {
    "TrainingObjective": "EmbeddingsOnly"
  }
}

Use Cases:

• TokenizerOnly — Build vocabulary for external tools
• EmbeddingsOnly — Generate raw token embeddings
• SentenceEmbeddings — Train model for semantic search (Contrastive Learning). Unlike Normal LLMs that predict the next word for generation, Sentence Embeddings compress full sentences into meaning vectors to measure similarity between them.
• PretrainOnly — Create checkpoints for continued training
• FullModel — Production-ready normal LLM models with full training (predicting the next word).

If not specified, defaults to FullModel (no breaking changes to existing configs).

Getting Started

1. Clone the repository
   bashCopy

   git clone https://github.com/magnolima/PascalGPT/PascalGPT.git

2. Open Demos/pascalgpt.trainer.dpr in Delphi 12+ and build the project (Release mode recommended).
3. Run the demo with a dataset:
   bashCopy

   pascalgpt.trainer.exe [-c config.json] [-train config.train.json] [-listgpu] dataset.txt

   • -c: Main configuration file (model params, paths).
   • -train: Training configuration overrides (epochs, batch size).
   • -listgpu: Lists available CUDA GPU devices and their IDs, then exits.
   • dataset.txt: Path to your training corpus.

You can select a specific GPU by defining DefaultGPUIndex inside the "General" section of config.json.

GPU Requirements

For CUDA acceleration:

• NVIDIA GPU with Compute Capability 6.0+ (Pascal or newer)
• CUDA Toolkit installed (for PTX compilation)
• nvcuda.dll available in system PATH

The framework automatically falls back to CPU if CUDA is unavailable.