README

Tiny NGRAM (LLM) Evaluation (Delphi / FMX)

A small study project that implements an N-gram language model (character- or word-based) with a FireMonkey (FMX) GUI to train on text files and generate continuations from a prompt. The UI is titled “NGRAM Model Evaluation” and is meant for quick experimentation with model order, smoothing, and sampling settings [1][2].

What this code does

• Creates an N-gram model with configurable:
  • Order (N) (e.g., 3–6 are typical)
  • Gram type: word or char [2][4]
  • Smoothing: Laplace, WittenBell, WittenBellFast [2][4]
  • Optional preserve case flag [2][4]
• Loads training text (from selected files/folders in the GUI), then trains the model by collecting counts of “context → next token” transitions (classic N-gram) [4].
• Generates text from a user seed/prompt by repeatedly sampling the next token using weighted probabilities [4].
• Includes Top‑P (nucleus) filtering to restrict sampling to the smallest set of tokens whose cumulative probability mass reaches the threshold (e.g., 0.9) [3].

Project layout (key units)

• GUI.FMX.TestLLMs.fmx / GUI.FMX.TestLLMs.pas
  FMX application + controls (gram type, smoothing selection, status/output memo) and model creation configuration [1][2].
• Unit_Math.AI.Ngram.pas
  Core TNGramModel implementation (ngChar/ngWord, smoothing options, generation/sampling logic) [4].
• Unit_Math.AI.Ngram.helpers.pas
  Helper utilities, including ApplyTopP(...) for nucleus sampling [3].

How to use (GUI)

1. Open the project in Delphi (FMX).
2. Run the app. You should see the “NGRAM Model Evaluation” window [1].
3. Choose:
   • Order (N) (e.g., 3)
   • Gram type: word or char [2]
   • Smoothing: Laplace / WittenBell / WittenBellFast [2]
   • Optional: Preserve Case [2]
4. Click Create Model to instantiate TNGramModel with your settings [2].
5. Load training text (via folder/file selection in the UI), then Train Model.
6. Type a seed prompt, set generation parameters (length / TopP etc. as exposed by your UI), and click Execute Model to generate output.

Notes on sampling

• The generator samples next tokens from weighted candidates (probability-like weights) [4].
• Top‑P filtering keeps only the highest-probability tokens until the cumulative mass reaches TopP [3].
• Smoothing affects how probabilities are assigned when contexts are sparse/unseen (smLaplace, smWittenBell, smWittenBellFast) [4].

How to extend

1) Add a new smoothing method

1. Extend the TSmoothing enum (currently includes smLaplace, smWittenBell, smWittenBellFast) [4].
2. Implement the corresponding probability/backoff logic inside the model where next-token weights are computed.
3. Add the new option to the GUI combo box initialization (currently adds Laplace/WittenBell/WittenBellFast) [2].

2) Add new sampling controls (Top‑K, temperature, etc.)

• Top‑P already exists as a helper (ApplyTopP) [3].
• You can add:
  • Top‑K: sort candidates by weight, keep K
  • Temperature: transform weights with p_i^(1/T) (or logit scaling if you convert to logits)
• Then call these filters/transforms in the token selection step used during generation [4].

3) Improve tokenization

• For ngWord, you can refine splitting rules (punctuation handling, normalization, Unicode categories).
• For ngChar, you can optionally support multi-byte grapheme clusters instead of code units.

4) Persistence / interoperability

• There are GUI buttons for saving/loading; you can extend the model with:
  • Versioned serialization
  • Export to JSON for inspection/debugging
  • Compression for large count tables

Limitations (expected for a study project)

• N-grams capture local context only; coherence degrades with longer text.
• Training data quality strongly affects output.
• Large orders and word-based models can grow memory quickly.

---

Delphi 11 Compiler