README

syc

A streaming archiver and compressor with its own LZ codecs, written from scratch in Free Pascal, with C and assembler for the hottest loops. Tuned for modest hardware.

[ciBadge]: https://github.com/YadeWira/syc/actions/workflows/ci.yml/badge.svg "Continuous Integration" [ciLink]: https://github.com/YadeWira/syc/actions/workflows/ci.yml [relBadge]: https://img.shields.io/github/v/release/YadeWira/syc?include_prereleases "Latest release" [relLink]: https://github.com/YadeWira/syc/releases

CI builds and tests every platform above and checks they all compress to the same bytes, also with the Pascal-only build (-dSYLZ_PUREPASCAL).

Status: v0.5.x, pre-release. Every release stays a pre-release until v1.0.0. syc v0.1.x was a Rust archiver; its sources are archived in legacy/rust/. v0.2 restarted the project in Free Pascal.

Goals

"Modest hardware" is taken literally:

• bounded memory, small match-finder tables, reusable contexts;
• good on 1–4 cores rather than needing 16;
• no AVX2 requirement: x86-64 baseline (SSE2) is enough;
• 32-bit targets supported (the Pascal code paths run anywhere FPC does); on 32-bit x86 the decoder picks its SSE2 version at run time (cpuid) where the CPU has SSE2, its i686 one elsewhere.

Languages: Free Pascal first; C and x86-64 assembler only where Free Pascal's code falls short, each next to a Pascal version that stays the reference (the tests check they give the same bytes). C never becomes the main language.

sylz

The first codec: a fast LZ with separate control and literal streams, so the decoder finds the next sequence at a fixed distance whatever the literal length, and a token table derived from measured parses, so that few sequences need a length extension (a branch the CPU can't predict). Format v3 (-ec, extra compression, experimental) adds Huffman-coded literals, in 64 KiB chunks with four interleaved streams; a v2 block is a valid v3 block. Eleven levels, -0 to -10 (default -1): -0 and -1 are greedy single-hash levels (-1 with the bigger table); -2 and -3 use two hash tables, keyed on 6 and 4 bytes (-2 greedy, -3 lazy); -4 to -8 hash chains with lz4hc's parse (matches may start inside the previous one and extend backwards); -9 and -10 an optimal parser over a binary-tree match finder, whose prices include decode time, so it picks cheaper sequences. The decoder is safe on corrupt input. Hot loops have x86-64 assembler versions (Linux and Win64) next to the Pascal ones, the optimal parser's DP a C one, and the fuzzer checks that every path produces the same bytes.

silesia.tar, in memory, one block, one core (Xeon E5-2697A v4, 2.6 GHz), everything in one session (2026-10-23; the machine was moderately busy, load ~4, so speeds vary by ~10%). Fast and middle levels:

High ratios:

On very redundant data -10 searches deeper than -9 and pays for it: zpaq-std's mixed sample 17.73% (-9) -> 17.59% (-10).

32-bit Windows and arm64 have no assembler: there the C paths run. Windows 7 32-bit (a VM, mozilla): compress -1 282 MB/s, -4 66.6 MB/s, -8 18.6 MB/s, decode 1640-2370 MB/s; the Pascal paths made 177, 39.8 and

11. MB/s, decode 850-1150 MB/s. Benchmarks.

Format v3 (-ec, Huffman-coded literals, experimental), every level, in another session (load ~4): -ec -0 .. -8 parse as -2 .. -10 do (v2's -0 and -1 leave so many literals that, Huffman-coded, they decode slower and compress worse than -2's parse); -ec -9 / -10 run the optimal parser with each literal priced by its Huffman code (two parses).

Same session, v2 -10: 36.03% at 3498 MB/s decode. With one thread v3 decodes at 60-65% of v2's speed on silesia.tar, at twice zstd -1's; on mixed data zstd -1 is still smaller.

Competitors measured with lzbench 2.4.1 (GCC 14.2, no -march=native) on the same core. More, and how: Benchmarks. The format: sylz format.

Build

Free Pascal 3.2.2 and a C compiler for the target (gcc; gcc -m32 for Linux x86; clang on macOS; mingw-w64 for Windows; aarch64-linux-gnu-gcc for arm64; CC overrides), which build.sh picks from the compiler's target:

./build.sh                    # out/syc and out/sylzbench
./build.sh -Twin64            # Windows x86-64 (FPC's win64 units;
                              #   with FPC's Windows installer: -Px86_64 -Twin64)
./build.sh -dSYLZ_PUREPASCAL  # no assembler, no C: Pascal paths only
FPC=ppcross386 ./build.sh -Twin32                  # Windows 32-bit
FPC=ppcrossa64 ./build.sh -XPaarch64-linux-gnu-    # Linux arm64
scripts/fpc-i386.sh /opt/fpc-i386                  # Linux x86: a 32-bit FPC
FPC=/opt/fpc-i386/fpc ./build.sh                   #   (needs gcc-multilib)

On macOS (brew install fpc) ./build.sh builds as on Linux.

Release binaries: Linux x86-64, arm64 and x86, Windows x86-64 and 32-bit, macOS arm64. The macOS ones are not signed: after downloading, run xattr -d com.apple.quarantine syc (or allow it in System Settings › Privacy & Security). Setting up the cross compilers: Building.

Use

Every command and option: Usage (and syc h CMD). The archive (.syc, format 2) and stream (.sylz, version 4) formats are frozen since v0.4.0: syc archive format, syc stream format.

out/syc a backup.syc docs/ photos/   # a new archive: deduplicated, solid blocks
out/syc a backup.syc docs/ photos/   # again: a new version with what changed
out/syc l -l backup.syc              # list (filters: paths, directories, globs)
out/syc i backup.syc                 # versions, sizes, deduplication
out/syc t backup.syc                 # check everything, write nothing
out/syc x backup.syc -o restore/     # extract (-f replaces, --skip leaves existing files)
out/syc x -u 1 backup.syc -o old/    # as of version 1 (or a date: -u 2026-10-24)
out/syc l --all backup.syc docs/     # every version of each file
out/syc a -p secret.syc docs/        # encrypted (asks for the password; or --password-file)
out/syc k secret.syc                 # a new password, in place (no recompression)

out/syc c -1 file          # -> file.sylz   (-0 .. -10, -ec, -b MIB, -t N threads)
out/syc d file.sylz        # -> file        (keeps the input; --rm deletes it)
out/syc t file.sylz        # check it, write nothing
tar c dir | out/syc c > dir.tar.sylz       # no FILE: stdin -> stdout
out/syc h a                # every option of a command
out/sylzbench -l 1,9 FILE  # in-memory benchmark, lzbench-style

An existing output is not overwritten without -f. Exit status: 0 ok, 1 an input skipped, 2 usage, 3 corrupt data, 4 I/O error.

Each block carries a CRC32 of its data, checked on decompression.

zpaq-std plugins

plugin/ holds one plugin file for zpaq-std (plugin ABI 1) with two codecs: sylz (-ma:sylz, format v2) and sylzec (-ma:sylzec, format v3: -ec's), levels 0..10, default 1. It exports zpaqstd_codec_v1 (sylz, what zpaq-std 1.7 and earlier load) and zpaqstd_codecs_v1(i) (0 sylz, 1 sylzec, then NULL). Before v0.4.2 sylzec was a file of its own.

plugin/build.sh linux-x86_64          # out/plugin/linux-x86_64/sylz.so
plugin/build.sh win64                 # .../win64/sylz.dll
FPC=ppcross386 plugin/build.sh win32 -Fu<rtl>/units/i386-win32
FPC=ppcrossa64 plugin/build.sh linux-arm64 -Fu<rtl>/units/aarch64-linux \
  -Fl/usr/lib/gcc-cross/aarch64-linux-gnu/14 -Fl/usr/aarch64-linux-gnu/lib

plugin/test_plugin.c loads the plugin the way zpaq-std does (--codec N picks a codec of the list) and checks the contract: exported entry, ABI fields, roundtrips at every level, compress never past cap (also when cap is too small), corrupt and truncated streams, 8 threads at once, and that loading the library leaves the process's floating-point state (MXCSR, x87 control word) alone. It passes on Linux x86-64 and arm64 (qemu-user), Windows 10 x64 and Windows 7 (32-bit plugin).

sylz writes format v2 as frozen at tag v0.2.2, sylzec format v3 as frozen at tag v0.2.14 (a v3 block, or plain v2 when that is smaller); a later format would be a new codec name in zpaq-std. For decoders written elsewhere (zpaq-std's ZPAQL ones): tools/sylz_ref.py, a reference decoder in Python written from the format description, and tests/sylzvectors.pas, which writes test vectors (released as sylz-vectors.zip).

Test

fpc -O3 -Fusrc -FUout/units -FEout tests/sylzfuzz.pas
out/sylzfuzz 3000 SAMPLE-FILES...

Roundtrips random, structured and real data at every level, decodes corrupted and truncated blocks against guard-page buffers (any read or write out of bounds faults), alternates the assembler and Pascal paths, and checks the two compressors agree byte for byte. Unix only (uses mmap). .github/test.sh runs every test the way CI does.

Layout

src/           sylz codec (sylz.pas, and sylzc.c: its C part), its Huffman
               coder (sylzhuf.pas), the syc CLI (syc.pas)
bench/         sylzbench, sylzstats (sequence histograms of a block)
tests/         sylzfuzz, sylzedge, sylzvectors (test vectors for other decoders)
tools/         sylz_ref.py: a reference decoder in Python
plugin/        zpaq-std codec plugin and its test
build.sh       (and build-c.sh, which compiles the C part for the target)
NOTES.md       development log: measurements, FPC codegen findings
legacy/rust/   syc v0.1.x, the Rust archiver: README, NOTES and its sources
               archived in syc-rust-v0.1.29.zip

Documentation

The wiki: format, benchmarks, building, Free Pascal codegen notes, versioning, and the legacy Rust version.

License

MIT, see LICENSE.

The legacy Rust v0.1.x was published as MIT OR Apache-2.0; its archive in legacy/rust/ also holds third-party code (packJPG, packPNG, ppmd-rust) under those projects' own licenses.