## Summary
Phase #2 of the `.swb` step-cache size-reduction roadmap (after #2465, v5 uniform elision). Stores the 16 base directional Z arrays as masked residuals against each cell's own SourceZ and omits any array that matches its prediction. Lossless, byte-identical reconstruction.
Trammel: 231.9 MB → 124.7 MB (−46%).
## Details
- Predictor: `predict = mask bit ? SourceZ : 0` (matches the baker's 0 on unwalkable directions); residual `Z − predict` via unchecked two's-complement (byte-exact for all inputs); reconstruct `Z = predict + residual`.
- A `u16 ZArrayMask` flags which of the 16 base arrays differ from prediction; matching arrays are omitted and synthesized from mask + SourceZ at read.
- Serializer-layer only: StepChunk, the cache, the algorithm, and the baker are unchanged.
- Format v6; v5 files rejected and re-baked once.
## Tests
21 v6 unit tests; full pathfinding suite green; Release build clean.
## Summary
Sub-project #1 of the `.swb` step-cache size-reduction roadmap (`dev-docs/pathfinding.md` § Future work). Adds **uniform-chunk elision** to the `StepCacheFile` format, bumping it **v4 → v5**.
A fully-uniform 16×16 chunk — no strata, **no swim layer**, all 19 base arrays constant (open ocean, Green Acres, void) — serializes to a **~28-byte record** (`KindUniform`) instead of ~5,393, and reconstructs **byte-identically** via `Array.Fill`. Non-uniform chunks use the existing v4 body (`KindFull`) with the swim-layer and strata trailers **fully preserved** — the Kind byte is just prepended.
## Calibrated result (measured, not projected)
Baked Trammel via `SaveToFile`:
| | |
|---|---:|
| Chunks | 114,688 |
| Uniform (swim-aware) → elided | 62.7% |
| Swim-layer chunks (stay Full) | 8.9% |
| Strata chunks (stay Full) | 1.8% |
| Baseline (full records) | 592.2 MB |
| **Actual v5 `.swb`** | **231.9 MB (−61%)** |
The residual is ~150 MB of non-uniform land Z-blocks (targeted by #2 predictive-Z) + ~81 MB of swim-layer trailers (#2/#3). #2 and #3 are separate follow-up PRs.
## Implementation
- `StepChunk.IsUniform()` — false if it has strata **or a swim layer**, else true only when all 19 base arrays are constant (the "all-same" check uses the SIMD-accelerated `ContainsAnyExcept`).
- `StepCacheFile` v5 — `Kind` byte (`KindFull=0`/`KindUniform=2`, 1 reserved); uniform write/read; `FormatVersion`/`MinSupportedVersion` → 5 (v4 files rejected on open and re-baked). No `StepCache`/algorithm/index changes; fingerprint logic untouched.
## Tests
7 `StepCacheFileV5Tests` (uniform round-trip + `<200 B` compactness, varied-full, swim-layer-full, strata-full, swim+strata combined, v4 version-gate rejection) + the existing StepCache/pathfinding suite — **70 pass**, including the prior `SwimLayer_RoundTrips`. An independent review verified write/read symmetry, cast round-tripping, swim/strata preservation, and the version gate (READY TO MERGE).
## Summary
Closes the Cold-cache regression flagged in PR #2450. `StepCache.TryGetMask` no longer eagerly runs `BuildChunk` on the first miss for a chunk that isn't in a `.swb` lazy reader. Instead it returns `Fallthrough_NotBuilt` and the caller (`BitmapAStarAlgorithm`) takes the per-cell slow path. The chunk is only promoted to the bitmap fast path after the **second** miss within a 30-second window, filtering single-touch pass-throughs.
This makes BitmapAStar's worst-case (cold cache + short hops) collapse from **12–47× slower** than FastAStar to **roughly the same**, which is the floor the slow path can deliver. Steady-state warm performance (the actual deliverable) is unchanged from PR-5 — it was always the cache fast path.
## The pet-follow scenario this fixes
A mounted player at ~4 tiles/sec with a pet/hireable following will trigger an NPC pathfind every 100–300 ms. Each pathfind is 1–6 tiles. As the player crosses chunk boundaries (~4 sec/chunk), the pet's first pathfind in the new chunk under the previous behavior triggered a full ~700 µs `BuildChunk` for a chunk the player would leave shortly after. At 50–100 mobiles per shard, this exceeded the 8 ms tick budget. PR-5 BDN data showed scenarios 6–9 (2–8 tile NPC perception) at 2,300–3,700 µs Cold vs FastAStar's 80–200 µs.
Under the new gate:
- First miss → `Fallthrough_NotBuilt` → caller uses slow path (~30–50 µs short path). No `BuildChunk`. No allocation.
- Player keeps moving → chunk never gets a second touch within window → never promoted, no rot.
- NPC patrolling a fixed territory → repeatedly hits the same chunks → second touch within window → promote → cache fast path on subsequent calls.
## What changed
- **`CacheHitKind.Fallthrough_NotBuilt = 6`** + **`CacheStats.FallthroughNotBuilt`** counter. `IsHit=false`, so the caller routes to slow path.
- **`StepCache._chunkMissTracker`** — `Dictionary<long, ChunkMissState>` capped at 4096 entries. State is `(byte missCount, uint lastMissTickStamp)` keyed by chunk key. Window-expired entries reset count to 1; capacity overflow prunes window-old entries first.
- **`StepCache.MissPromotionThreshold`** (default `2`) and **`StepCache.MissPromotionWindowMs`** (default `30_000`) — tunable, can be wired through `ServerConfiguration` if shards want different policy. Setting threshold to `1` restores legacy eager-build behavior (used by tests that prime chunks via single `TryGetMask` call).
- **`StepCache.TryGetMask` miss branch** — try lazy reader first (file-loaded chunks bypass the tracker entirely; an `.swb` represents an explicit prior decision to keep the chunk warm). Otherwise consult the tracker.
- **`BitmapAStarAlgorithm.GetSuccessorsSlowPath`** now layers `IsBlockedByDynamic` on top of `CalcMoves.CheckMovement`. Previously the slow path only ran for `CanFly` creatures and rare cache fallthroughs — `CheckMovement` doesn't iterate same-cell mobiles, so the bitmap fast path's `IsBlockedByDynamic` was the only mobile-blocking check. Now first-touch pathfinds run through the slow path, so the gap had to close.
## Tests
50 pathfinding tests pass (was 47). New / updated:
- **`TryGetMask_FirstTouchOnUnbuiltChunk_DefersBuildAndReturnsFallthrough`** — single TryGetMask call returns `Fallthrough_NotBuilt`, no chunk built, no allocation.
- **`TryGetMask_SecondTouchWithinWindow_PromotesAndBuilds`** — second call inside the 30s window builds + serves.
- **`TryGetMask_SecondTouchAfterWindow_RestartsCounterAndDefers`** — second call outside the window restarts the count, returns Fallthrough again.
- **`TryGetMask_DistinctChunks_TrackedIndependently`** — counters are per-chunk; one touch on each of two adjacent chunks both stay in fallthrough.
- **`LazyReaderHit_BypassesMissTrackerOnFirstTouch`** — open `.swb` + first touch hits without consulting the tracker. Production with `.swb` loaded skips the gate entirely.
- **`MultisVersion_Bump_TriggersDirtyRebuild`** — updated to reflect the new 3-step flow (Fallthrough → Miss_NotBuilt → Miss_DirtyRebuild).
- Tests that prime chunks via a single `TryGetMask` call (multi-Z, Tier4, lifecycle, parity, BitmapAStar uses-cache) set `MissPromotionThreshold = 1` to opt into eager behavior.
## Expected BDN impact
The Cold column from PR-5's BDN should change as follows once the bench's submodule pointer is updated to this branch:
| # | Scenario | Cold (PR-5) | Cold (PR-6 expected) | FastAStar Cold |
|--:|-----------------|-------------:|---------------------:|---------------:|
| 2 | sewer corridor | 1,627 µs | ~36 µs | 36 µs |
| 4 | causeway | 1,533 µs | ~39 µs | 39 µs |
| 6 | pet 2-tile | 2,364 µs | ~80 µs | 81 µs |
| 8 | npc 5-tile | 3,708 µs | ~140 µs | 141 µs |
| 9 | npc 8-tile | 2,386 µs | ~200 µs | 197 µs |
WarmNoFile and LazyWarm rows should be unchanged — they were always cache-warm. The miss tracker only fires when neither resident chunks nor the lazy reader can satisfy the request.
## Future work (not in this PR)
- **Background-thread bake**: builds outside the game thread so even promoted chunks don't pay the 700 µs build cost on the main thread. Rule 10 (no Task.Run) applies, so this needs careful design — the bake is a pure data transform but main-thread synchronization on chunk-state transitions has to be threaded through. Defer to a follow-up.
- **Long-traverse BDN scenario**: a multi-Find benchmark simulating 50 pet repaths across chunk transitions. Requires restructuring the bench harness; the existing 10-scenario corpus + Cold provider already exercises the gate.
- **Swim sourceZ bake**: scenario 5 (sea serpent) shows 56 B alloc on warm paths because the cache's SourceZ is computed under default-walker rules. Swim creatures fall through to slow path. Independent of this PR.
## Summary
Multi-Z cells (bridges, stairs, paver-over-ground, multi-floor structures) now carry **per-stratum walkability data** in the cache instead of falling through to the slow path. The data is computed at chunk-build time, persisted in the `.swb` file, and selected at query time by matching the request's `sourceZ` against each stratum's `zCenter` (within `StepHeight` tolerance).
This is the Tier 4 strata feature, deferred from PR #2447 / PR #2448 / PR #2449. Builds on PR #2449's lazy backing store and public bake helpers.
## Wire format change (v1 → v2)
`StepCacheFile.FormatVersion = 2`. `MinSupportedVersion = 2`. v1 `.swb` files are silently rejected at open time (treated as missing) and overwritten on the next `SaveToFile` / `BakeMap`. **No migration** — older files just get re-baked.
The `MinSupportedVersion` sentinel is the model going forward: bump the constant when an incompatible change lands; admins re-bake on the next deploy. No matrix of v1↔v2↔v3 migration logic to maintain.
## What changed
- **`StepProbe.ComputeStrataAt(map, x, y)`** — enumerates walkable standing-Zs at the cell (one per land surface plus one per walkable static), collapses Zs within `2*StepHeight`, runs `ComputeMaskAt` at each surviving Z. Returns `null` for single-Z cells (caller uses the chunk's main mask).
- **`StepChunk`** — replaces the old `MultiZCells` bitmap with a **strata storage pair**:
- `ushort[256] StrataOffsetByCell` (sentinel `NoStrata = 0xFFFF` = "no strata for that cell")
- `byte[] StrataData` packed: `u8 stratumCount`, then `count × 19-byte stratum`
- `sbyte zCenter, byte walkMask, byte wetMask, sbyte walkZ_N..NW (8), sbyte swimZ_N..NW (8)`
- `IsCellMultiZ` derives from `StrataOffsetByCell[cell] != NoStrata` — same semantics, single source of truth.
- **`StepCache.BuildChunk`** — populates strata for cells flagged multi-Z via `SetStrata`. Chunks with zero multi-Z cells pay zero strata overhead (offset array + data array stay null).
- **`StepCache.TryGetMask`** — for multi-Z cells, scans strata with `TryStratumHit`; returns the matching one with `HitKind=Hit`. Falls through to slow path only when no stratum matches the query `sourceZ`.
- **`StepCacheFile`** — v2 serialization with strata trailer per chunk + `recordLength` in index entry. Lazy reader sizes scratch per-chunk-record using the recorded length, growing on demand for multi-Z-heavy chunks. Patches `IndexOffset` on `w.Buffer` (BufferWriter's current backing array) since it grows during variable-size chunk writes.
## File layout v2
```
Header (48 bytes):
u32 Magic = 0x42575300 ('SWB\0')
u32 Version = 2
u32 MapId
u64 Fingerprint XxHash3 over LandTable + ItemTable flags + map files (mapX.mul/.uop, staidxX.mul, staticsX.mul)
u64 BakeTimestamp informational
u32 ChunkCount
u64 IndexOffset position where chunk index begins
Per chunk (variable size):
u16 ChunkX, ChunkY
u32 BuiltMultisVersion
u8 HasStrata 0 = no strata trailer; 1 = strata trailer follows
byte WalkMask[256], WetMask[256]
sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256]
// Strata trailer (only when HasStrata == 1):
u16 StrataOffsetByCell[256] // NoStrata sentinel = 0xFFFF
u32 StrataDataLength
byte StrataData[StrataDataLength]
Per multi-Z cell: u8 count, then count × Stratum (19 bytes)
Index trailer (20 × ChunkCount bytes):
per chunk: { u64 chunkKey, u64 fileOffset, u32 recordLength }
```
## Summary
Adds two pieces of pathfinding tooling on top of PR #2448's lazy `.swb` infrastructure:
- **`PathfindRecorder`** — admin-toggled JSONL telemetry capture; one record per `BitmapAStarAlgorithm.Find` call. Output format matches the BDN harness corpus, so production traffic can be captured and replayed in benchmarks without an adapter.
- **Public bake helpers on `StepCache`** — `ComputeLiveTileDataHash`, `TryReadTileDataHashFromFile`, `BakeMap`, `ClearResidentChunks`. Lets the benchmark project (and any future bake utility) drive cache fill + persist without exposing internal types.
The companion BDN harness update lives in [ModernUO-Benchmarks#kb/pathfinding-pr4-bench](https://github.com/modernuo/ModernUO-Benchmarks/tree/kb/pathfinding-pr4-bench): porting `Benchmarks/PathfindInGame/` from the `kb/ai_pathfinding` branch to the API shipped in #2446–#2448.
## What's in this PR
### `PathfindRecorder` (`PathfindRecorder.cs`)
- Holds a single `StreamWriter` open while recording; its internal buffer absorbs per-record writes without per-call `File.AppendAllText`.
- Single `bool` check on the hot path; cheap when disabled.
- Disabling flushes + disposes; an IO failure during write also disables the recorder.
- Server config:
- `pathfinding.recorder.enable` — bool, default `false`. Read on boot via `GetOrUpdateSetting`.
- `pathfinding.recorder.path` — default `<basedir>/Data/Pathfinding/recordings/pathfinds.jsonl`.
- Hooked into `BitmapAStarAlgorithm.Find` — runs once per call, does nothing when disabled.
- Admin command: `[PathRecord [on|off|flush|status]` (default `status`).
### Public cache helpers
- `static ulong StepCache.ComputeLiveTileDataHash()` — wraps the file module's hash function for staleness checks.
- `static bool StepCache.TryReadTileDataHashFromFile(string, out ulong)` — peeks at a `.swb` file's hash field (20 bytes).
- `int StepCache.BakeMap(int, string)` — walks every chunk in the map, populates resident set, saves. Offline / fixture use; blocks for many seconds on a full-map walk.
- `void StepCache.ClearResidentChunks()` — drops chunks + zeros counters but keeps lazy readers open. Lets benchmark loops measure "first query after boot" cost across iterations without the lazy-reader reopen overhead.
## Summary
Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them.
The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all.
Builds on PR #2447.
## What changed
- **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary.
- **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them.
- **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity.
- **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map.
- **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime.
- **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector).
- **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly.
- **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map.
- **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow.
- **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings.
## File layout (v1)
```
Header (48 bytes):
u32 Magic = 0x42575300 ('SWB\0')
u32 Version = 1
u32 MapId
u64 TileDataHash XxHash3 over LandTable + ItemTable flags (HashUtility)
u64 BakeTimestamp informational
u32 ChunkCount
u64 IndexOffset file position where the chunk index begins
Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z):
u16 ChunkX
u16 ChunkY
u32 BuiltMultisVersion
u8 HasMultiZ
byte WalkMask[256], WetMask[256]
sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256]
[byte MultiZCells[32] when HasMultiZ == 1]
Index trailer (16 × ChunkCount bytes):
(u64 chunkKey, u64 fileOffset)
```
## Memory math
| Scenario | Disk file | RAM at boot | Notes |
|---|---|---|---|
| Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. |
| Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. |
| Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. |
| All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. |
## Hash choice (FNV-1a → XxHash3)
The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`:
- ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere.
- Stronger collision resistance and distribution.
- Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern.
- Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
Kamron Batman