README

Suto — A Lox Dialect in Delphi Pascal

Suto is a bytecode-interpreted, garbage-collected, NaN-boxed dialect of Bob Nystrom's Lox language, implemented in Delphi Pascal — extended with a 2D game engine, event-driven callbacks, file I/O, SQL database access, records, and peephole optimizations. The syntax is Lox; the runtime, extensions, and native surface are ours.

The VM follows Bob's Crafting Interpreters (Chapters 14–26) and then keeps going.

Why "Suto"?

Suto (Tongan) — to hitch a ride; a freeloader; someone who tags along without contributing. Used affectionately among friends: "Oi, suto!"

This project is a suto on Bob's work. Every core concept in the VM — the scanner, the Pratt parser, the bytecode chunk, the stack machine, the closure representation, the mark-sweep GC, the hash table — came from Crafting Interpreters. We took the ride. The extensions on top (records, arrays, dictionaries, NaN boxing, game engine, RTTI injection, event dispatcher, SQL) are ours, but the frame they hang on is Bob's.

The name is chosen with the same tone the word carries in Tongan: half acknowledgement, half self-mockery, and above all a nod of respect to whoever's doing the driving. Nystrom's book is one of the most generous gifts to hobby language design in the last decade. Naming the language Suto is us tipping our cap.

Features

Language & VM

• Scanner & Compiler — Single-pass Pratt parser compiling to bytecode
• Virtual Machine — Stack-based VM with call frames, closures, and upvalues
• NaN Boxing — TValue = UInt64; doubles stored as raw IEEE-754 bits, non-number values encoded as quiet NaN payloads. Canonicalizes NaN inputs via CANON_NAN to prevent tag-space collisions. ~2.6× faster than tagged-union on fib(40).
• Garbage Collection — Mark-sweep GC with tricolor marking and gray stack worklist
• String Interning — Hash table with weak references for automatic deduplication
• Dynamic Stack — Growable value stack (starts at 256 slots, grows to 65 536) with automatic frame/upvalue pointer rebasing after realloc
• Arrays — Dynamic arrays with literal syntax ([1, 2, 3]), subscript get/set (arr[i]), plus native functions (newArray, arrayPush, arrayPop, arrayGet, arraySet, arrayLen, arrayRemove)
• Dictionaries — Hash-table dictionaries with generalized keys (any hashable TValue), Swift-style literal syntax ([:], ["key": val]), subscript get/set, PascalCase method API (Set, Get, Has, Delete, Keys, Values, Size), linear probing with tombstones, 75% load factor, bounded probing, GC-integrated
• Records — Immutable-structure value types via record Name(field1, field2); syntax with dot access and field mutation
• Native Objects — Inject Delphi classes as GC-tracked Lox objects with automatic RTTI-based property and method dispatch
• RTTI Injection — Any Delphi TObject descendant can be injected into a Lox script via InjectObject(); published properties and methods are automatically accessible without wrapper code. Supports nested objects, object-typed method arguments, sets, dynamic arrays, property mutation, method invocation with marshaling between Lox and Delphi types. Denylist prevents scripts from calling Free/Destroy and other lifecycle methods.
• Anonymous Functions — First-class lambda expressions: fun(params) { body } usable anywhere an expression is valid — assigned to variables, passed as arguments, returned from functions, or immediately invoked
• Callbacks — InvokeCallback re-entrant VM dispatch allowing Delphi natives to call Lox closures; callWith(fn, arg) and invoke(fn, ...) variadic natives for host-to-script function invocation

Performance Optimizations

• NaN Boxing — Eliminates heap-allocated tagged unions; all values fit in a single 8-byte register
• Peephole Optimizations — Fused opcodes: OP_JUMP_IF_FALSE_POP (fused test+pop), OP_LESS_JUMP_IF_FALSE (fused comparison+branch for for loops), OP_POP_N (bulk pop)
• Fast Local Slots — OP_GET_LOCAL_0..7 / OP_SET_LOCAL_0..7 encode slot index in the opcode itself, saving 1 bytecode byte and 1 dispatch per access
• Inline Fast Paths — pushStack / popStack / peekStack are inline with cold-path delegation for stack growth
• NaN Canonicalization — CreateNumber uses Move() (not pointer punning) for aliasing-safe bit conversion; NaN payloads that collide with tag space are canonicalized to CANON_NAN

2D Game Engine

• Software-rendered canvas — 320×240 logical resolution (configurable), auto-scaled with letterboxing, double-buffered
• Sprites — Create from pixel arrays, PNG files, PNG regions, or palette indices; draw, scale, rotate, flip; automatic sprite sheet slicing
• Tilemaps — Create tile grids from sprite sheets; set/get individual tiles; draw with camera offset
• Surfaces — Off-screen render targets for compositing
• Camera — Global camera offset for scrolling
• Input — keyHeld(), keyPressed(), mouseX/Y(), mouseDown(), mouseClicked() — edge-triggered per frame
• Sound — 4-channel waveform synthesizer: playNote(freq, duration, waveform, channel), playSequence() for melodies, stopSound(), stopAllSound()
• Game Loop — processMessages() pumps the Windows message queue mid-script; scripts run as cooperative game loops with event polling
• Event Engine — Decoupled TLoxEventEngine with callback-driven input: onKeyPressed, onKeyReleased, onKeyHeld, onMouseDown, onMouseUp callbacks invoked per-frame via processEvents(); held-key tracking with snapshot-based dispatch to prevent re-entrancy crashes

File I/O & Process Execution

• File Operations — fileExists(path), directoryExists(path), readFile(path), writeFile(path, content), writeCsv(path, rows) (RFC 4180 compliant)
• Process Execution — exec(cmd) returns stdout as string; execResult(cmd) returns dictionary with output, exitCode, success fields — non-zero exit is not a runtime error

Database Access

• SQL Server — sqlConnect(config), sqlQuery(conn, sql), sqlQueryParams(conn, sql, params), sqlClose(conn) — parameterized queries returning arrays of dictionaries, with connection pooling via FireDAC
• Environment Variables — env(name), loadEnv() for .env file loading (connection strings, secrets)

IDE & Tooling

• Syntax-Highlighted Editor — SynEdit-based Script Pad with custom Lox highlighter (keywords, built-ins, strings, numbers, comments), dark theme, line numbers, auto-indent, group undo
• Event Queue — TLoxQueue native object enables Delphi-to-Lox event passing; Delphi code enqueues string events from form handlers, Lox scripts poll via events.hasItems() / events.dequeue()
• Interactive Script Execution — processMessages() enables interactive scripts; automatic print output flushing; script abort on form close
• Script Runtime Safety — Run button, test buttons, and editor are disabled while a script is running
• Modular Native Registry — Each native domain lives in its own unit (Natives/*.pas); NativeRegistry.RegisterAllNatives auto-discovers and registers all modules at startup
• VM Introspection — vmStackDepth(), vmStackCapacity(), vmCallDepth(), vmOpenUpvalues(), gcNextThreshold(), gcCollectionCount(), internTableStats(), loxClasses(), loxClassInfo(), loxObjects(), loxObjectInfo()
• Long-Operand Opcodes — 24-bit constant indices for globals, closures, literals, and dot access (16M constant limit)

Chapters Implemented

Extensions Beyond the Book

Native Functions (110+)

Project Structure

Suto.pas                 ??? Core unit: scanner, parser, compiler, VM, GC, hash table
LoxCanvas.pas            ??? 2D game engine: canvas, sprites, tilemaps, surfaces, input
LoxSound.pas             ??? 4-channel waveform sound synthesizer
LoxEventEngine.pas       ??? Reusable event-dispatching engine (key/mouse callbacks)
NativeObjects.pas        ??? Native Delphi classes exposed to Lox (TLoxQueue, TCustomer, TAddress)
fmGame.pas / fmGame.dfm  ??? Game window: hosts canvas, input, event engine, script lifecycle
fmEventTest.pas          ??? Lightweight event-engine test form
Main.pas / Main.dfm      ??? Editor form with interpreter REPL, test runner, syntax highlighting
SynHighlighterLox.pas    ??? Custom SynEdit Lox syntax highlighter
InterpreterGui.dpr       ??? Delphi project file

Natives/                 ??? Native function modules (one unit per domain):
  NativeRegistry.pas     ??? Central registration system (RegisterAllNatives)
  SystemNatives.pas      ??? clock, assert, GC, VM introspection, env
  ArrayNatives.pas       ??? Array manipulation natives
  DictNatives.pas        ??? Dictionary manipulation natives
  StringNatives.pas      ??? String library natives
  MathNatives.pas        ??? Math library natives
  ConversionNatives.pas  ??? str, num, bool, type
  IntrospectionNatives.pas ??? loxClasses, loxObjects, internTableStats
  SqlNatives.pas         ??? SQL Server access via FireDAC
  FileNatives.pas        ??? File I/O: read, write, CSV export
  ProcessNatives.pas     ??? exec, execResult (child process execution)
  EventNatives.pas       ??? Event callback natives + simulation natives
  CallbackNatives.pas    ??? callWith, invoke (host-to-script function calls)

docs/                    ??? Architecture documentation (11 chapters)
samples/                 ??? Custom Lox test programs (auto-run on startup)
samples/demos/           ??? Game demos (Space Invaders, Manic Cavern, Mario, slots, etc.)
samples/tests/events/    ??? Event engine callback tests (27 files)
samples/errors/          ??? Expected-error test programs
samples/stress/          ??? GC stress patterns (run via Button2)
samples/introspection/   ??? VM introspection scripts (GC monitor, stack probe, dashboard)
samples/Profiler/        ??? Performance benchmarks
test/                    ??? Official Crafting Interpreters test suite + extensions
test/native/             ??? RTTI injection tests
test/record/             ??? Record type tests (23 files)

NaN Boxing

All Lox values are represented as a single UInt64 (TValue):

Safety measures:

• NaN Canonicalization — CreateNumber detects NaN payloads that collide with tag space and maps them to CANON_NAN ($7FF8000000000000)
• Pointer Validation — CreateObject raises ELoxRuntimeError at runtime if a pointer has bits in the tag region (detects platform incompatibility)
• Aliasing-safe conversion — Move() used instead of pointer type-punning for Double↔UInt64 bit conversion

Testing

All tests run automatically when the application starts. Results are displayed in Memo2. You can also type Lox code into the Script Pad and click Run.

Official Test Suite

156+ tests from the official Crafting Interpreters test suite, matching the chap26_garbage level (all features except classes/inheritance), plus custom dictionary, array, record, and NaN boxing tests.

The test runner parses // expect:, // expect runtime error:, and // [line N] Error comments from each .lox file and verifies actual output matches expected results.

Custom Sample Tests

46 samples in samples/ — expected to pass (INTERPRET_OK):

Event Engine Tests

27 tests in samples/tests/events/ — callback-driven input system:

Game Demos

21 demos in samples/demos/ — interactive game scripts:

GC Stress Test Suite

1 comprehensive stress file in samples/stress/ — run via the Run Stress Tests button:

15 patterns: allocation storm, long/short-lived mix, pointer graphs, cycles, mutation, deep chains (2000 levels), container thrash, random lifetimes, closure capture, combined stress, shared-object explosion, sustained pressure (5000 iterations), cycle reclamation verification, intern table churn (2000+ strings), stack realloc with open upvalues.

Performance Benchmarks

11 benchmark scripts in samples/Profiler/ covering arithmetic, arrays, branching, closures, dictionaries, function calls, GC stress, globals vs locals, constant loading, records, and strings.

RTTI Injection Test Suite

21 tests in test/native/ — RTTI-based native object injection, run via the Run RTTI Tests button:

Happy-path tests (property access, methods, nesting, closures, equality, GC safety, arrays, dicts, functions, loops) plus 11 error tests (undefined property, denylist, wrong arity, bad arg types).

GC Hardening

• DEBUG_STRESS_GC — Triggers CollectGarbage on every growth allocation
• DEBUG_LOG_GC — Logs allocate/free/mark/blacken events to gc.log with summary stats
• DEBUG_STRESS_TABLE — Enables AssertTableConsistency on every table operation
• Push/pop protection at all GC-sensitive allocation sites
• NextGC floor of 1024 bytes to prevent zero-threshold assertions
• Stack realloc rebase — After ReallocMem moves the stack buffer, rebases all VM.Frames[i].slots and open upvalue location pointers
• Stack overflow guard — Hard limit at 65 536 slots with ELoxRuntimeError raise (not silent corruption)
• 16 dedicated GC test files + 15-pattern stress suite + NaN boxing edge cases

Build

Prerequisites

• Delphi (RAD Studio 12+) — Win32 target
• SynEdit — Install via GetIt Package Manager (Tools → GetIt Package Manager → search "SynEdit")

Building

1. Open InterpreterGui.dproj in the Delphi IDE
2. Ensure the target platform is Win32 and configuration is Debug or Release
3. Build (Shift+F9) or Run (F9)

Notes

• SynEdit is an IDE design-time package — it must be installed into the IDE, not just added to the search path
• The custom Lox syntax highlighter (SynHighlighterLox.pas) is included in the project
• FireDAC (included with RAD Studio) is used for SQL Server connectivity
• No other external dependencies are required