feat(pathfinding): .swb format v7 per-chunk compression (#3a) (#2470)

## Summary
Phase #3a, stacked on #2469. Compresses each chunk record independently with libdeflate (random access preserved).

Trammel: 124.7 MB → 19.2 MB (−85%).

## Details
- Whole-record framing `[u32 UncompressedLen][payload]`; records that don't shrink (tiny Uniform) are stored raw, detected as payload length == UncompressedLen.
- Codec chosen by full-Trammel spike: libdeflate VeryHigh (16.5 MB, 1.83 µs/chunk decompress) over zstd L19/22 (17.4 MB) and managed Brotli q11 (16.4 MB) — best native ratio, fastest decompress, already the repo's packet codec (no new dependency).
- Reuses cached thread-static bindings: `Deflate.Maximum` for bake, `Deflate.Standard` for reads.
- Compression is bake-time only; decompression is one-time per chunk (LRU-cached).
- Format v7; v6 files rejected and re-baked once.

## Tests
v7 unit tests + the v6 suite run through the compression path; full pathfinding suite green; Release build clean.
This commit is contained in:
Kamron Batman 2026-06-07 00:22:11 -07:00 committed by GitHub
parent 94537f83f8
commit c265bcbb5e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 255 additions and 15 deletions

View file

@ -221,11 +221,20 @@ whole-file) and bounded RAM (only touched chunks materialize, LRU-capped):
saves ~0 bytes (a residual is still 1 byte/cell); the win is the per-array elision, and leaving
non-elided arrays in residual form makes #3 a pure codec add (no further transform/format bump).
Swim-layer + strata trailers stay absolute, deferred to #3.
3. **Per-chunk block compression + index compaction.** zstd/deflate each chunk record
independently (index already carries a per-chunk `length`; reader decompresses one chunk on
read). At 256-cell granularity the **index overhead** then dominates (20 B/chunk ×
114 K ≈ 2.3 MB), so compact it in the same pass: chunks in fixed sweep order → implicit keys,
delta-encoded offsets, lengths derivable → ~24 B/chunk.
3. **Per-chunk block compression + index compaction.**
- **#3a — per-chunk compression: *shipped as format v7.*** Each record is libdeflate-compressed
independently (random access preserved; the reader inflates one chunk on read into a reused
buffer) behind a `u32 UncompressedLen` frame; tiny Uniform records that do not shrink are
stored raw (detected as on-disk payload length == UncompressedLen). Codec chosen by full-Trammel
spike: **libdeflate VeryHigh** beat zstd L19/22 (17.4 MB) and tied managed Brotli q11 (16.4 MB)
at **16.5 MB of compressed records**, with the *fastest* decompress (**1.83 µs/chunk**, vs zstd
2.21) — and it is already the repo's packet codec (`LibDeflate.Bindings`), so no new dependency.
zstd's large-window advantage doesn't apply at 256-cell record granularity. Compression runs at
bake time only (offline); decompression is one-time per chunk (LRU-cached after). **Calibrated:
v6 124.7 MB → v7 19.2 MB (85%); 97% vs the original 565 MB.**
- **#3b — index compaction (remaining, ~2 MB).** The v7 file's residual is the still-uncompacted
index (20 B/chunk × 114 K ≈ 2.3 MB). Compact it: chunks in fixed sweep order → implicit keys,
delta-encoded offsets, lengths derivable → ~3 B/chunk ≈ 0.34 MB, landing the file at ~17 MB.
**Do first:** a uniformity audit over a baked facet (how many chunks are fully uniform / have
all-zero Z residuals?) to size the #1/#2 win before writing any format code. Each technique is a
@ -251,3 +260,8 @@ Validate size **and** read-latency vs the corpus in the benchmark repo after eac
by present walk-residual blocks (~46 MB, mostly ±1..3 + zeros → highly compressible), the
per-Full-chunk mask/SourceZ base (~33 MB), and absolute swim-layer trailers (~26 MB) — all prime
targets for #3 per-chunk compression.
- *Calibrated v7 (`BakeMap`-measured, full Trammel):* **#3a per-chunk libdeflate VeryHigh: 124.7 MB
→ 19.2 MB (85%); 97% vs the original 565 MB.** Codec spike (per-chunk over the 122.4 MB of v6
records): libdeflate VeryHigh **16.5 MB** vs Brotli q11 16.4 / zstd L1922 17.4; decompress
**1.83 µs/chunk** (libdeflate) vs 2.21 (zstd). The remaining ~2.3 MB is the uncompacted index
(→ #3b).