warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

perf(pathing): pool the StepCache strata buffer, then clean up the pathing engine around it (#2523)

Browse source 
Started as an allocation pass over `StepCache` and grew into a cleanup of the surrounding pathing engine. Four commits, each independently reviewable; net **−560 lines**.

Build clean (0 warnings). All 122 `Server.Tests.Pathfinding` tests pass.

---

## 1. `perf`: pool the strata buffer, cut a hot-path dictionary lookup

**The headline is that `TryGetMask` — the actual hot path — was already allocation-free.** `StepMask` is a readonly struct, `StaticTileEnumerable` is a `ref struct`, `ChunkMissState` is a struct in a `Dictionary`. So most of this is a bake-throughput and GC-churn win, with one exception noted below.

`BuildChunk` accumulated packed multi-Z strata into a `List<byte>` that grew by doubling (256 → 512 → 1024 → …) and then paid a final `ToArray()`. A full map bake runs it ~114k times. It now writes into a `byte[]` rented from `STArrayPool<byte>.Shared` through a span writer, and hands the chunk one exact-size copy.

**This required fixing a latent out-of-bounds guard.** The record-fit check reserved headroom for **8** strata (`StratumByteLength * 8`) while `ComputeStandableSurfaceZs` can return up to **16** — so a cell could write 305 bytes starting from a 65,383-byte offset. Against a `List` that was benign (it just grew past 64 KB, and emitted offsets stayed under the `NoStrata` sentinel). Against a fixed-size rented buffer it is an out-of-bounds write, so tightening it was a *prerequisite* for the pooling, not a drive-by. The guard is now exact, which additionally proves no emitted offset can collide with `NoStrata == ushort.MaxValue`.

**One genuine query-path win:** `ShouldPromoteAfterMiss` did *two* dictionary lookups per miss — a `TryGetValue`, then an indexer assignment that re-hashes and re-probes. It now mutates in place via `CollectionsMarshal.GetValueRefOrNullRef`. This runs on every uncached chunk touch during A* expansion. The window-expiry branch keeps its explicit early return, so `MissPromotionThreshold == 1` still resets rather than promoting.

Also dropped `StepProbe.ComputeStrataAt` / `ComputedStratum` (dead code, zero callers) and collapsed six 18-argument `new StepMask(0, 0, …, kind)` blocks into `Fallthrough(kind)`.

**Considered and rejected:** pooling the `Direction[]` that `Find` returns. It *escapes* the call — `MovementPath` holds it across ticks while `PathFollower` walks `m_Index` through it — so it cannot be rented-and-returned, and it cannot be borrowed from the shared `BitmapAStarAlgorithm.Instance` without one creature clobbering another's in-flight path. `CheckPath` rate-limits repaths to one per 2s per creature, putting this at roughly 60 KB/sec at 1,000 pathing creatures. Not worth a public API break plus a use-after-return footgun.

## 2. `docs`: rewrite the comments for publication

The comments had accumulated as development notes: internal phase jargon (`Tier 4`, `the Phase-2 synthesizer`), change narration aimed at a reviewer (`which the old ComputeStandingZ anchor missed`, `legacy behavior`), benchmark anecdotes (`benchmarked as near-optimal`, `a ~20 ns lookup`), and paragraphs restating the code.

Rewritten to keep the rationale you cannot recover by reading the code — why the source-Z guard cannot be widened, why multis fall through with a halo, why the promotion gate counts Finds rather than calls, why `ComputeFingerprint` must hash the *files* and not the live tile tables — and drop the history that got us there.

Three comments were **factually wrong**, not just wordy:

- `CacheEvictionTimer` and `CacheStats` documented a class called `StaticWalkabilityCache`. No such class exists — it is `StepCache`.
- `StepCacheFile` declared `File layout v8` while `FormatVersion` is 9, and called the current record layout "the v6 layout" in four places. The layout descriptions are now unversioned so they cannot drift again.
- `StepProbe.ComputeStandingZ` claimed `StepCache` uses it to bake `SourceZ`. It has not since the baker moved to the clearance-aware `ComputeStandableSurfaceZs`; only a parity test calls it.

## 3. `refactor`: simplify `StepCacheFile.Write`, consolidate the format tests

`SaveToFile` walked `_keysList` **twice** — once to count the map's chunks, then again through a `ChunkEnumerator` closure to emit them — because `Write` needed the count up front to size its index array. Both loops had the same root cause. Passing a **span** collapses them: the count is just `span.Length`.

That deletes the `ChunkEnumerator` delegate, the closure over the list enumerator, and **both `InvalidOperationException` throws**, which existed only to police the delegate's "yield exactly `chunkCount` chunks" contract — a contract a span makes unrepresentable.

`Write` now patches the header's `IndexOffset` by seeking back to it rather than reaching into the writer's live buffer with `BinaryPrimitives`. That also retires `IndexOffsetFieldPosition`, a hand-maintained byte offset that had to track the header layout, and sidesteps the stale-array hazard that motivated the manual patch (`BufferWriter` reallocates on growth).

**Tests:** `StepCacheFileV6/V7/V8Tests` were named for the format version that introduced each transform — and the format is now **v9**, so all three names described formats the loader rejects outright. Beyond triplicated builders and plumbing, two things were actually broken:

- The three near-identical rejection tests each cited a `MinSupportedVersion` that had since moved (`"version 5 < MinSupportedVersion 6"`, `"6 < 7"`, `"7 < 8"`). They passed for the wrong reason.
- `AssertBaseEqual` (used by V7 and V8) **silently skipped the swim and strata trailers**. A regression dropping either would not have failed those tests.

Now one `StepCacheFileFormatTests`, named for behavior — predictive-Z elision, compression, compact index — with a single `AssertIdentical` that does check both trailers, the three rejection tests folded into one theory that also covers a future version, and a zero-chunk case the delegate-based writer never had coverage for.

## 4. `test`: consolidate the parity and lifecycle tests

Three files tested "parity" and none of the names said *which*. They were three different layers, and the seams are the useful part, so they are now one `StepCacheParityTests` that names them:

| Test | Compares | Answers |
|---|---|---|
| `ProbeMatchesSlowPath` | StepProbe vs MovementImpl | Is the bake right? |
| `CacheMatchesProbe` | StepCache vs StepProbe | Is it stored and returned intact? |
| `CacheServesReachableWalkStates` | StepCache vs MovementImpl | End to end, over the states A* visits |

Merging removed a duplicated stub `Mobile`, duplicated region seeds, and a filename/class mismatch (`StepProbeParityTests.cs` declared `StaticWalkabilityParityTests`). `SwimBake_ProducesWetCells` moved with it — it lived in the cache parity file but never touched the cache.

Tests reached into `StepCache._chunks` via `GetField` in **9 places**, each rebuilding the key encoding and cell-index arithmetic by hand. `StepCache` now exposes `GetResidentChunk` and `ResidentIndexInSync` alongside the internal test hooks it already had (`LazyReaderHasChunk`, `CurrentFindGeneration`), and the shared arithmetic moved to `PathingTestSupport`. All 9 reflection blocks are gone.

`StepCacheLifecycleTests` is regrouped by what it covers — promotion gate, fallthrough routes, strata, swim layer, eviction — with the `Tier4*` names dropped. Removed `Singleton_IsAvailable`, which asserted an inline-initialized static property was not null; that is the entire 123 → 122 test-count delta.

---

## Verification

Tests were mutation-checked rather than just run, since round-trip and parity tests can pass while a transform silently no-ops:

- Injecting an off-by-one into the `IndexOffset` patch fails **15 of 123** — the format tests are load-bearing.
- Offsetting the cache's cell index by one fails **7 of 10** parity cases, and the 3 that stay green are exactly the ones that do not touch the cache. The layering localizes a fault rather than just reporting one.
This commit is contained in:
Kamron Batman 2026-07-12 20:02:29 -07:00 • committed by GitHub
parent e035768ef8
commit b852bca41e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 1663 additions and 2223 deletions
Download patch file Download diff file Expand all files Collapse all files

297
Projects/UOContent/Engines/Pathing/Cache/StepCacheFile.cs
View file

@ -9,40 +9,36 @@ using Server.Compression;
namespace Server.Engines.Pathing.Cache;
/// <summary>
/// Binary serializer + lazy reader for the step cache. Persists chunk records to disk
/// so a server warm-starts without paying chunk-build cost on the first pathfind through
/// a region. Lazy: opening a file reads only the header + chunk-offset index (~few KB
/// for tens of thousands of chunks), then individual chunks are seeked + deserialized
/// only when the cache asks for them. RAM stays bounded by MaxResidentChunks regardless
/// of file size.
/// Binary serializer and reader for the step cache, so a shard can warm-start instead of building
/// chunks on the first pathfind through each region. Opening a file reads only the header and chunk
/// index; a chunk record is seeked and inflated when the cache actually asks for it, which keeps
/// resident memory bounded by MaxResidentChunks no matter how large the file is.
///
/// File layout v8 (little-endian, BufferWriter / BufferReader convention):
/// File layout (little-endian, BufferWriter / BufferReader convention):
///
/// Header (40 bytes):
/// u32 Magic = 0x42575300 ('SWB\0')
/// u32 Version = current FormatVersion (9)
/// u32 Version = FormatVersion
/// u32 MapId
/// u64 Fingerprint XxHash3 over (1) LandTable + ItemTable flags AND (2) the
/// on-disk bytes of mapX.mul / .uop, staidxX.mul, staticsX.mul.
/// Rejects a load when EITHER tile flags shifted (client patch)
/// OR the map data was rewritten (CentredSharp / UOFiddler edit).
/// The .mul format has no built-in CRC; this is the only way
/// to detect those mutations.
/// u64 BakeTimestamp DateTime.UtcNow.Ticks at write time (informational).
/// u64 Fingerprint XxHash3 over tiledata.mul and the map's own .mul / .uop files.
/// Detects both a client patch that shifts tile flags and a map edit
/// that rewrites the terrain; see ComputeFingerprint. The .mul format
/// carries no CRC of its own, so hashing is the only way to catch either.
/// u64 BakeTimestamp DateTime.UtcNow.Ticks at write time. Informational.
/// u32 ChunkCount
/// u64 IndexOffset File position where the chunk index begins.
/// u64 IndexOffset Where the index trailer begins.
///
/// Per chunk (ChunkCount times, variable size):
/// u32 UncompressedLen Size of the inflated record body below.
/// byte[] Payload The record body (the v6 layout that follows), libdeflate-
/// compressed. If the on-disk payload length (index recordLength −
/// 4) equals UncompressedLen, the body was stored raw because
/// compression did not shrink it (tiny Uniform records).
/// byte[] Payload The record body, libdeflate-compressed — or stored raw when
/// compression didn't shrink it, as happens with tiny Uniform
/// records. The reader tells the two apart by comparing the payload
/// length against UncompressedLen.
///
/// Record body (after inflate — the v6 layout):
/// Record body (after inflate):
/// u16 ChunkX
/// u16 ChunkY
/// u32 BuiltMultisVersion (reserved since v9 — always 0; chunks are static-only)
/// u32 BuiltMultisVersion Reserved, always 0 — chunks are static-only.
/// u8 Kind 0 = Full; 2 = Uniform
/// // Uniform (Kind == 2): ~28-byte record — all 256 cells share these single values:
/// byte walkMask, wetMask; sbyte sourceZ; sbyte walkZ_N..NW (8); sbyte swimZ_N..NW (8)
@ -78,34 +74,29 @@ namespace Server.Engines.Pathing.Cache;
/// The file offset is not stored — reconstructed as a cumulative sum of recordLength
/// starting at HeaderSize (the first record sits immediately after the header).
///
/// Per-chunk fixed portion (Kind + flags + ZArrayMask + WalkMask + WetMask + SourceZ):
/// ~783 bytes; each present base Z array adds 256 bytes (0..16 present, so up to ~4 KB).
/// A fully-flat Full chunk stores no residual blocks. Strata trailer: 516 + N × ~30 bytes
/// for a chunk with N multi-Z cells averaging ~2 strata each. LRU bookkeeping
/// (LastTouchedTicks) is intentionally not persisted.
/// A chunk's fixed portion runs ~783 bytes, and each directional-Z array that survives prediction
/// adds 256 more, so a Full record lands between ~783 bytes and ~4 KB. The strata trailer adds
/// 516 bytes plus roughly 30 per multi-Z cell. LastTouchedTicks is deliberately not persisted —
/// LRU state means nothing across a restart.
///
/// Files with version &lt; <see cref="MinSupportedVersion"/> are silently rejected
/// at open time (treated as missing) and overwritten on the next save.
/// Files below <see cref="MinSupportedVersion"/> are treated as missing and overwritten on the
/// next save. The cache regenerates from the map data, so a format bump only costs a one-time
/// re-bake.
/// </summary>
internal static class StepCacheFile
{
public const uint Magic = 0x42575300; // 'SWB\0'
// v9: chunks are STATIC-ONLY (land + statics.mul, no multis). v8 and earlier baked multis
// (houses/boats) into chunks, which is unsafe to persist — multis are dynamic, and the
// BuiltMultisVersion they were tagged with is a non-persisted session counter. Bumping the
// version rejects those old files so they re-bake static-only. The BuiltMultisVersion record
// field is retained as a reserved (always-0) u32 to avoid a layout change.
public const uint FormatVersion = 9;
/// <summary>
/// Lowest format version this binary can load. Files below it are treated as missing
/// (silently rejected) and overwritten by the next SaveToFile / BakeMap. The cache is
/// fully regenerable, so a format bump just forces a one-time re-bake of stale files.
/// Oldest format this binary will load. Anything older is treated as missing rather than
/// migrated: the cache is fully regenerable from the map data, so a re-bake is always
/// available and always correct.
/// </summary>
public const uint MinSupportedVersion = 9;
// Per-chunk record discriminator (first byte after BuiltMultisVersion). 1 is reserved.
// Record discriminator. 1 is reserved.
private const byte KindFull = 0;
private const byte KindUniform = 2;
@ -118,12 +109,12 @@ internal static class StepCacheFile
+ sizeof(uint) // ChunkCount
+ sizeof(ulong); // IndexOffset
// Index entry (v8 compact): u32 packedKey ((chunkX << 16) | chunkY) + u32 recordLength.
// The file offset is NOT stored — entries are in record write order, so the reader
// reconstructs each offset by cumulative sum of record lengths starting at HeaderSize.
// One index entry: u32 packedKey ((chunkX << 16) | chunkY) + u32 recordLength. The file offset
// isn't stored — entries sit in record write order, so the reader rebuilds each offset as a
// running sum of the lengths before it, starting at HeaderSize.
private const int IndexEntryBytes = sizeof(uint) + sizeof(uint);
/// <summary>Fixed-size portion of a chunk record (everything except the optional strata + swim trailers).</summary>
/// <summary>A chunk record minus its optional strata and swim trailers.</summary>
private const int BytesPerChunkBase =
sizeof(ushort) + sizeof(ushort) + sizeof(uint)
+ sizeof(byte) + sizeof(byte) + sizeof(byte) // Kind + HasStrata + HasSwimLayer
@ -135,17 +126,8 @@ internal static class StepCacheFile
+ 8 * StepChunk.CellsPerChunk; // SwimZ[8]
/// <summary>
/// Byte offset of the IndexOffset u64 within the header
/// (Magic+Version+MapId+Fingerprint+BakeTimestamp+ChunkCount = 32). Patched after chunks land.
/// </summary>
private const int IndexOffsetFieldPosition = 32;
public delegate bool ChunkEnumerator(out int chunkX, out int chunkY, out StepChunk chunk);
/// <summary>
/// Peek at a .swb file's Fingerprint field (header byte offset 12) without
/// reading any chunk data. Returns false on missing file, bad magic, or wrong
/// version. Cheap — reads 20 bytes total.
/// Reads just a .swb file's fingerprint — 20 bytes, no chunk data. False if the file is
/// missing, isn't a .swb, or is a version this binary can't load.
/// </summary>
public static bool TryReadFingerprint(string path, out ulong fingerprint)
{
@ -183,31 +165,28 @@ internal static class StepCacheFile
}
/// <summary>
/// Combined XxHash3 fingerprint over (1) the on-disk <c>tiledata.mul</c> file and (2) the
/// per-map .mul / .uop file contents (via <see cref="TileMatrix.MapFilesFingerprint"/>).
/// Bake files carry this hash so a load can refuse to populate the cache when EITHER the
/// tile data shifted (client patch) OR the map data was rewritten (CentredSharp / UOFiddler
/// edit). The .mul format has no built-in CRC; this is the only way to detect those mutations.
/// Hashes the inputs a bake depends on: <c>tiledata.mul</c> and the map's own .mul / .uop
/// files. A file carrying a stale hash is refused at open time, which is what catches a client
/// patch that shifts tile flags or a map editor that rewrites the terrain. Neither format has a
/// CRC of its own, so hashing is the only signal available.
///
/// IMPORTANT: hash the FILES, never the in-memory <see cref="TileData.LandTable"/> /
/// This must hash the FILES, never the in-memory <see cref="TileData.LandTable"/> /
/// <see cref="TileData.ItemTable"/>. The server patches those tables at runtime (ItemFixes,
/// LOSBlocker, PotionKeg, CTF, ...) at nondeterministic lifecycle points, so a fingerprint over
/// the live tables varies with WHEN it is taken; the file hash is the only lifecycle-stable
/// "did the client's tile data change?" signal. Server-side tile patches are applied identically
/// every boot and intentionally do NOT invalidate the cache — change one and you must
/// [PathCacheClear or bump the format.
/// LOSBlocker, PotionKeg, CTF), so a hash of the live tables changes depending on when it is
/// taken — useless as a fingerprint. Those server-side patches apply identically every boot and
/// deliberately do NOT invalidate the cache; if you change one, run [PathCacheClear or bump
/// <see cref="FormatVersion"/> yourself.
/// </summary>
public static ulong ComputeFingerprint(int mapId)
{
var hasher = HashUtility.CreateXxHash3();
// (1) tiledata.mul — hashed once, cached. The authoritative source for tile flags/heights.
Span<byte> tileDataBytes = stackalloc byte[sizeof(ulong)];
BinaryPrimitives.WriteUInt64LittleEndian(tileDataBytes, TileDataFileFingerprint());
hasher.Append(tileDataBytes);
// (2) Map files (mapX.mul / .uop, staidxX.mul, staticsX.mul). TileMatrix already
// streamed them through XxHash3 once at construction; mix the result in.
// TileMatrix already streamed the map files through XxHash3 when it was built; reuse that
// rather than re-reading them.
var map = Map.Maps[mapId];
if (map != null && map != Map.Internal && map.Tiles != null)
{
@ -223,10 +202,9 @@ internal static class StepCacheFile
private static bool _tileDataFileFingerprintComputed;
/// <summary>
/// XxHash3 over the raw <c>tiledata.mul</c> bytes, computed once and cached — the file never
/// changes during a run. Mirrors <see cref="TileMatrix.MapFilesFingerprint"/> for the map
/// files. Returns 0 if the file can't be found (the server can't run without it anyway, so
/// this only matters in stripped test hosts, where 0 is a fine deterministic constant).
/// XxHash3 of the raw <c>tiledata.mul</c> bytes, computed once — the file can't change while
/// the server runs. Returns 0 when the file is absent, which only happens in stripped test
/// hosts; a real server can't boot without it, and 0 is a fine deterministic stand-in.
/// </summary>
private static ulong TileDataFileFingerprint()
{
@ -249,84 +227,68 @@ internal static class StepCacheFile
}
/// <summary>
/// Writes the file: header (with placeholder IndexOffset) → chunks (offsets recorded)
/// → index trailer → patches the header IndexOffset. <paramref name="chunkCount"/> must
/// equal the actual number of chunks <paramref name="next"/> will yield.
/// Writes the map's chunks to <paramref name="path"/>: header, then one record per chunk, then
/// the index trailer. IndexOffset isn't known until the records are down, so it goes in as a
/// placeholder and gets patched by seeking back to it.
/// </summary>
public static void Write(string path, uint mapId, uint chunkCount, ChunkEnumerator next)
public static void Write(string path, uint mapId, ReadOnlySpan<(int chunkX, int chunkY, StepChunk chunk)> chunks)
{
Directory.CreateDirectory(Path.GetDirectoryName(path) ?? ".");
// Initial estimate: base record + a modest strata budget per chunk. Coastline
// chunks add another ~2.5 KB (swim layer) but they're a small fraction of any
// map; the writer grows on overflow so under-estimating just causes a few
// realloc/copy cycles during the bake — not a correctness issue.
var capacity = HeaderSize + (BytesPerChunkBase + 256) * (int)chunkCount + IndexEntryBytes * (int)chunkCount;
var buffer = new byte[capacity];
var w = new BufferWriter(buffer, prefixStr: false);
// A rough estimate: the base record plus a small strata budget per chunk. Coastline chunks
// run ~2.5 KB over it for their swim layer, but they're a small share of any map, and the
// writer grows on overflow — under-estimating costs a few reallocs during a bake, nothing more.
var capacity = HeaderSize + (BytesPerChunkBase + 256 + IndexEntryBytes) * chunks.Length;
var w = new BufferWriter(new byte[capacity], prefixStr: false);
w.Write(Magic);
w.Write(FormatVersion);
w.Write(mapId);
w.Write(ComputeFingerprint((int)mapId));
w.Write((ulong)DateTime.UtcNow.Ticks);
w.Write(chunkCount);
w.Write(0UL); // IndexOffset placeholder, patched after chunks
w.Write((uint)chunks.Length);
var indexOffsetPosition = w.Position;
w.Write(0UL); // patched below, once the records are written and the index position is known
// Each record is built uncompressed into recordScratch, then libdeflate-compressed into
// compScratch and framed as [u32 uncompressedLen][payload].
// Each record is built into recordScratch, compressed into compScratch, then framed as
// [u32 uncompressedLen][payload].
var packer = Deflate.Maximum;
var recordScratch = new byte[BytesPerChunkBase + 1024];
var compScratch = new byte[packer.MaxPackSize(recordScratch.Length)];
var indexEntries = new (ulong key, ulong offset, uint length)[chunkCount];
var written = 0u;
while (next(out var chunkX, out var chunkY, out var chunk))
// Record lengths only — the index stores no offsets, so the reader rebuilds them by
// summing these in order.
var lengths = new uint[chunks.Length];
for (var i = 0; i < chunks.Length; i++)
{
if (written >= chunkCount)
{
throw new InvalidOperationException(
$"StepCacheFile.Write: enumerator yielded more than the declared {chunkCount} chunks"
);
}
var chunkOffset = (ulong)w.Position;
var (chunkX, chunkY, chunk) = chunks[i];
var start = w.Position;
WriteChunk(w, chunkX, chunkY, chunk, packer, ref recordScratch, ref compScratch);
var chunkLength = (uint)((ulong)w.Position - chunkOffset);
indexEntries[written] = (PackChunkKey(chunkX, chunkY), chunkOffset, chunkLength);
written++;
}
if (written != chunkCount)
{
throw new InvalidOperationException(
$"StepCacheFile.Write: declared {chunkCount} chunks but enumerator yielded {written}"
);
lengths[i] = (uint)(w.Position - start);
}
var indexOffset = (ulong)w.Position;
for (var i = 0u; i < chunkCount; i++)
for (var i = 0; i < chunks.Length; i++)
{
// v8 compact entry: u32 packedKey ((chunkX << 16) | chunkY) + u32 recordLength.
// Offset is omitted; entries are in record write order so the reader derives it.
var key = indexEntries[i].key;
var packedKey = ((uint)(key >> 32) << 16) | (uint)(key & 0xFFFF);
w.Write(packedKey);
w.Write(indexEntries[i].length);
var (chunkX, chunkY, _) = chunks[i];
w.Write((uint)((chunkX & 0xFFFF) << 16 | chunkY & 0xFFFF));
w.Write(lengths[i]);
}
// Patch IndexOffset on the writer's current backing buffer (BufferWriter may
// have grown during chunk writes; the original `buffer` ref is stale after grow).
var liveBuffer = w.Buffer;
BinaryPrimitives.WriteUInt64LittleEndian(liveBuffer.AsSpan(IndexOffsetFieldPosition, 8), indexOffset);
var totalBytes = (int)w.Position;
File.WriteAllBytes(path, liveBuffer.AsSpan(0, totalBytes).ToArray());
w.Seek(indexOffsetPosition, SeekOrigin.Begin);
w.Write(indexOffset);
// w.Buffer, not the array handed to the constructor: BufferWriter reallocates on growth,
// which leaves that original reference pointing at a stale array.
File.WriteAllBytes(path, w.Buffer.AsSpan(0, totalBytes).ToArray());
}
/// <summary>
/// Opens a .swb file and reads only its header + chunk-offset index. Returns null on
/// missing file, magic / version mismatch, or Fingerprint mismatch (a stale bake
/// against a freshly patched client). Callers own disposal of the returned reader.
/// Opens a .swb file, reading only its header and chunk index. Null if the file is missing,
/// isn't a loadable .swb, or is a stale bake whose fingerprint no longer matches the live tile
/// and map data. The caller owns the returned reader.
/// </summary>
public static LazyReader OpenForLazy(string path)
{
@ -361,8 +323,6 @@ internal static class StepCacheFile
var version = BinaryPrimitives.ReadUInt32LittleEndian(headerBuf[4..]);
if (version < MinSupportedVersion || version > FormatVersion)
{
// Below the minimum supported version: treat as missing. Older files
// get silently overwritten on the next SaveToFile / BakeMap.
stream.Dispose();
return null;
}
@ -379,7 +339,7 @@ internal static class StepCacheFile
return null;
}
// Read the chunk-offset index in one shot.
// Pull the whole index in one read.
var indexBytes = (int)chunkCount * IndexEntryBytes;
var indexBuf = new byte[indexBytes];
stream.Position = (long)indexOffset;
@ -389,9 +349,8 @@ internal static class StepCacheFile
return null;
}
// v8 compact index: { u32 packedKey, u32 length } per chunk, in record write order.
// The file offset is not stored — reconstruct it by cumulative record length starting
// at the first record (immediately after the header).
// Entries are in record write order and carry no offset, so rebuild each one as a
// running sum of the record lengths, starting just past the header.
var offsets = new Dictionary<ulong, (ulong offset, uint length)>((int)chunkCount);
var runningOffset = (ulong)HeaderSize;
for (var i = 0; i < chunkCount; i++)
@ -416,27 +375,27 @@ internal static class StepCacheFile
private static ulong PackChunkKey(int chunkX, int chunkY) => ((ulong)(uint)chunkX << 32) | (uint)chunkY;
/// <summary>
/// Predicted directional-Z for one cell/direction: the cell's own SourceZ when the
/// direction is walkable/wet (mask bit set), else 0 — matching the baker, which leaves
/// non-walkable directional slots at their zero-initialized default
/// (StepProbe.ComputeMaskAt clears walkZs/swimZs and writes only on a successful step).
/// Guesses a cell's destination Z for one direction: on flat ground a step lands at the Z you
/// left from, so predict SourceZ where the direction is passable and 0 where it isn't. The
/// zero matches the baker, which only writes a slot on a successful step and leaves the rest
/// cleared. Most terrain is flat, so most predictions are exact and most residuals are 0 —
/// which is what makes the residual arrays compress away to nothing.
/// </summary>
internal static sbyte Predict(byte dirMaskByte, int bit, sbyte sourceZ) =>
(dirMaskByte >> bit & 1) != 0 ? sourceZ : (sbyte)0;
/// <summary>
/// Residual of an absolute directional-Z against its prediction. Unchecked two's-complement
/// so the transform is byte-exact for ALL sbyte inputs (no value-range constraint).
/// A destination Z's difference from its prediction. Wraps deliberately: two's-complement
/// round-trips exactly for every sbyte input, so no value range is off-limits.
/// </summary>
internal static sbyte EncodeResidual(sbyte z, sbyte predict) => unchecked((sbyte)(z - predict));
/// <summary>Inverse of <see cref="EncodeResidual"/>: absolute directional-Z = predict + residual.</summary>
/// <summary>Inverse of <see cref="EncodeResidual"/>.</summary>
internal static sbyte DecodeZ(sbyte predict, sbyte residual) => unchecked((sbyte)(predict + residual));
/// <summary>
/// The base directional-Z array for direction index d in canonical order: walk N..NW (0-7),
/// then swim N..NW (8-15). Index d uses WalkMask (d &lt; 8) or WetMask (d &gt;= 8) with
/// direction bit (d &amp; 7).
/// The destination-Z array for direction index d, in the canonical order the format stores them:
/// walk N..NW as 0-7, then swim N..NW as 8-15.
/// </summary>
private static sbyte[] GetBaseZArray(StepChunk c, int d) => d switch
{
@ -448,10 +407,9 @@ internal static class StepCacheFile
};
/// <summary>
/// Builds the uncompressed v6 record for one chunk into <paramref name="w"/>, libdeflate-
/// compresses it, and writes it framed as [u32 uncompressedLen][payload]. The payload is the
/// compressed bytes, or — when compression does not shrink the record (tiny Uniform records) —
/// the raw record itself; the reader distinguishes the two by payload length vs uncompressedLen.
/// Builds one chunk's record, compresses it, and frames it as [u32 uncompressedLen][payload].
/// When compression fails to shrink the record — as it does on the tiny Uniform ones — the raw
/// record is stored instead, and the reader tells the two apart by payload length.
/// </summary>
private static void WriteChunk(
BufferWriter w, int chunkX, int chunkY, StepChunk chunk,
@ -460,7 +418,7 @@ internal static class StepCacheFile
{
var rw = new BufferWriter(recordScratch, prefixStr: false);
BuildRecord(rw, chunkX, chunkY, chunk);
recordScratch = rw.Buffer; // may have grown; keep the larger buffer for reuse
recordScratch = rw.Buffer; // may have grown; hold onto the larger buffer for the next chunk
var recordLen = (int)rw.Position;
var bound = packer.MaxPackSize(recordLen);
@ -478,8 +436,8 @@ internal static class StepCacheFile
}
else
{
// Incompressible (or expanded): store the record raw. The reader detects this when
// the on-disk payload length equals the uncompressed length.
// Compression didn't help, so store the record raw. Payload length == uncompressedLen
// is how the reader recognizes that.
w.Write(recordScratch.AsSpan(0, recordLen));
}
}
@ -490,8 +448,8 @@ internal static class StepCacheFile
w.Write((ushort)chunkY);
w.Write((uint)chunk.BuiltMultisVersion);
// Kind: 0 = Full, 2 = Uniform. A uniform chunk (no strata, no swim layer, all 19 base
// arrays constant) stores one cell's worth of data (~28-byte record total).
// A uniform chunk — every cell identical — collapses to one cell's worth of data, ~28 bytes.
// Open water and solid rock make up a lot of a map, so this is worth the branch.
if (chunk.IsUniform())
{
w.Write(KindUniform);
@ -517,7 +475,7 @@ internal static class StepCacheFile
return;
}
w.Write(KindFull); // Full
w.Write(KindFull);
var strataOffsetByCell = chunk.GetStrataOffsetByCellForSerialization();
var strataData = chunk.GetStrataDataForSerialization();
@ -526,9 +484,9 @@ internal static class StepCacheFile
w.Write((byte)(hasStrata ? 1 : 0));
w.Write((byte)(hasSwimLayer ? 1 : 0));
// Predictive-Z: each base directional Z array is stored as a masked residual against
// SourceZ. Bit d of ZArrayMask is set only when array d differs from its prediction
// somewhere; cleared arrays are omitted and rebuilt from mask+SourceZ at read.
// Each destination-Z array is stored as residuals against its prediction (see Predict). An
// array that matches its prediction everywhere — the common case on flat terrain — is
// omitted entirely, and its ZArrayMask bit stays clear so the reader synthesizes it.
ushort zArrayMask = 0;
for (var d = 0; d < 16; d++)
{
@ -583,7 +541,6 @@ internal static class StepCacheFile
if (hasStrata)
{
// 256 × u16 offsets, then u32 length-prefixed strata byte array.
for (var i = 0; i < StepChunk.CellsPerChunk; i++)
{
w.Write(strataOffsetByCell[i]);
@ -600,7 +557,7 @@ internal static class StepCacheFile
private static StepChunk ReadChunk(byte[] buffer)
{
var r = new BufferReader(buffer);
// Skip ChunkX + ChunkY (already known via the index lookup).
// ChunkX + ChunkY — already known from the index lookup that got us here.
r.ReadUShort();
r.ReadUShort();
var multisVersion = (int)r.ReadUInt();
@ -608,7 +565,7 @@ internal static class StepCacheFile
var chunk = new StepChunk { BuiltMultisVersion = multisVersion };
if (kind == KindUniform) // Uniform — one cell's worth of the 19 base arrays, fill all 256 cells.
if (kind == KindUniform) // one cell's values, broadcast to all 256
{
Array.Fill(chunk.WalkMask, r.ReadByte());
Array.Fill(chunk.WetMask, r.ReadByte());
@ -640,8 +597,8 @@ internal static class StepCacheFile
r.Read(chunk.WetMask);
ReadSBytes(r, chunk.SourceZ);
// Predictive-Z reconstruction: present arrays carry residuals (z = predict + residual);
// absent arrays are synthesized from mask+SourceZ (z = predict, residual implicitly 0).
// Inverse of the write path: a stored array carries residuals to add back to the
// prediction, an omitted one IS the prediction.
Span<sbyte> residual = stackalloc sbyte[StepChunk.CellsPerChunk];
for (var d = 0; d < 16; d++)
{
@ -706,16 +663,15 @@ internal static class StepCacheFile
r.Read(MemoryMarshal.Cast<sbyte, byte>(arr.AsSpan()));
/// <summary>
/// Open handle on a .swb file. Holds the FileStream + chunk-offset index. Chunks are
/// fetched on demand via <see cref="TryReadChunk"/>; only the records actually queried
/// are ever materialized. Dispose releases the underlying stream.
/// An open .swb file: the stream plus the chunk index. Only the records actually asked for are
/// ever read or inflated. Dispose releases the stream.
/// </summary>
internal sealed class LazyReader : IDisposable
{
private FileStream _stream;
private readonly Dictionary<ulong, (ulong offset, uint length)> _offsets;
private byte[] _buffer; // raw on-disk record: [u32 uncompressedLen][payload]
private byte[] _bodyBuffer; // decompressed v6 record, parsed by ReadChunk
private byte[] _buffer; // the raw framed record as it sits on disk
private byte[] _bodyBuffer; // that record, inflated, ready for ReadChunk
public uint MapId { get; }
public ulong Fingerprint { get; }
@ -725,11 +681,7 @@ internal static class StepCacheFile
public bool Has(int chunkX, int chunkY) => _offsets.ContainsKey(PackChunkKey(chunkX, chunkY));
/// <summary>
/// Enumerates every (chunkX, chunkY) coordinate the file holds. Used by
/// <see cref="StepCache"/> when preload is enabled to materialize all chunks
/// upfront instead of on first query.
/// </summary>
/// <summary>Every (chunkX, chunkY) the file holds. Used to preload the whole file.</summary>
public IEnumerable<(int chunkX, int chunkY)> EnumerateChunkCoords()
{
foreach (var key in _offsets.Keys)
@ -754,9 +706,8 @@ internal static class StepCacheFile
}
/// <summary>
/// Returns the chunk record at (<paramref name="chunkX"/>, <paramref name="chunkY"/>)
/// from the file, or null if the file doesn't contain it. Single seek + bulk read,
/// sized exactly to the chunk's recorded length (which varies with strata size).
/// Reads one chunk from the file, or null if the file has no record for it. One seek and
/// one read, sized to the record's indexed length.
/// </summary>
public StepChunk TryReadChunk(int chunkX, int chunkY)
{
@ -771,7 +722,6 @@ internal static class StepCacheFile
return null;
}
// Grow the on-disk scratch buffer if this chunk's record is larger than what we have.
if (entry.length > _buffer.Length)
{
_buffer = new byte[entry.length];
@ -784,8 +734,8 @@ internal static class StepCacheFile
return null;
}
// Frame: [u32 uncompressedLen][payload]. payload is libdeflate-compressed, unless its
// length equals uncompressedLen, in which case it was stored raw (incompressible).
// [u32 uncompressedLen][payload], where the payload is compressed unless its length
// already equals uncompressedLen — then it was stored raw.
var uncompressedLen = (int)BinaryPrimitives.ReadUInt32LittleEndian(_buffer);
var payloadLen = (int)entry.length - sizeof(uint);
if (_bodyBuffer.Length < uncompressedLen)
@ -799,7 +749,8 @@ internal static class StepCacheFile
}
else
{
// Decompression is level-independent, so reuse the shared per-thread binding.
// Deflate.Standard, not .Maximum: the level only affects packing, and inflate has
// to accept whatever the writer produced regardless.
var result = Deflate.Standard.Unpack(
_bodyBuffer.AsSpan(0, uncompressedLen),
_buffer.AsSpan(sizeof(uint), payloadLen),