feat(pathfinding): lazy .swb backing store for the step cache (#2448)
## Summary Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them. The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all. Builds on PR #2447. ## What changed - **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary. - **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them. - **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity. - **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map. - **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime. - **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector). - **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly. - **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map. - **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow. - **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings. ## File layout (v1) ``` Header (48 bytes): u32 Magic = 0x42575300 ('SWB\0') u32 Version = 1 u32 MapId u64 TileDataHash XxHash3 over LandTable + ItemTable flags (HashUtility) u64 BakeTimestamp informational u32 ChunkCount u64 IndexOffset file position where the chunk index begins Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z): u16 ChunkX u16 ChunkY u32 BuiltMultisVersion u8 HasMultiZ byte WalkMask[256], WetMask[256] sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256] [byte MultiZCells[32] when HasMultiZ == 1] Index trailer (16 × ChunkCount bytes): (u64 chunkKey, u64 fileOffset) ``` ## Memory math | Scenario | Disk file | RAM at boot | Notes | |---|---|---|---| | Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. | | Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. | | Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. | | All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. | ## Hash choice (FNV-1a → XxHash3) The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`: - ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere. - Stronger collision resistance and distribution. - Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern. - Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
This commit is contained in:
Kamron Batman
GitHub
parent
9066e8fd00
commit
7c9215d97c
6 changed files with 861 additions and 4 deletions
|
|
@ -50,6 +50,22 @@ public static class HashUtility
|
|||
return result;
|
||||
}
|
||||
|
||||
public static ulong ComputeHash64(ReadOnlySpan<byte> bytes)
|
||||
{
|
||||
if (bytes.Length == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
var hasher = _xxHash3 ??= new XxHash3(unchecked((long)xxHash3Seed));
|
||||
hasher.Append(bytes);
|
||||
|
||||
var result = hasher.GetCurrentHashAsUInt64();
|
||||
hasher.Reset();
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
public static uint ComputeHash32(ReadOnlySpan<char> str)
|
||||
{
|
||||
if (str == ReadOnlySpan<char>.Empty)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,213 @@
|
|||
using System.IO;
|
||||
using Server.Engines.Pathing.Cache;
|
||||
using Xunit;
|
||||
|
||||
namespace Server.Tests.Pathfinding;
|
||||
|
||||
[Collection("Sequential Pathfinding Tests")]
|
||||
public class StepCacheFileTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Round-trip a populated cache through a .swb file under lazy loading: build chunks,
|
||||
/// save, clear (closes lazy readers + drops residents), open as lazy backing store,
|
||||
/// then re-query and verify chunks come back from the file (not the runtime baker).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void RoundTrip_LazyLoad_PreservesChunkMaskAndZ()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
Assert.NotNull(map);
|
||||
|
||||
// Populate three distinct chunks by querying different sectors.
|
||||
var sourceQueries = new[] { (1500, 1600), (1516, 1600), (1500, 1616) };
|
||||
foreach (var (x, y) in sourceQueries)
|
||||
{
|
||||
cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
Assert.Equal(3, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(3L, cache.GetStats().BuildsTotal);
|
||||
|
||||
// Snapshot the answers we expect to recover after round-trip.
|
||||
var expected = new StepMask[sourceQueries.Length];
|
||||
for (var i = 0; i < sourceQueries.Length; i++)
|
||||
{
|
||||
var (x, y) = sourceQueries[i];
|
||||
expected[i] = cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-roundtrip-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
var written = cache.SaveToFile(path, map.MapID);
|
||||
Assert.Equal(3, written);
|
||||
|
||||
cache.Clear();
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
|
||||
Assert.True(cache.TryOpenLazyReader(path, map.MapID));
|
||||
Assert.Equal(1, cache.OpenLazyReaderCount);
|
||||
|
||||
// Lazy: opening doesn't materialize chunks, so the resident set stays empty
|
||||
// until we query.
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal);
|
||||
|
||||
// Sanity: the lazy reader's index covers each chunk we're about to query.
|
||||
foreach (var (x, y) in sourceQueries)
|
||||
{
|
||||
Assert.True(cache.LazyReaderHasChunk(map.MapID, x >> 4, y >> 4),
|
||||
$"lazy reader missing chunk ({x >> 4},{y >> 4})");
|
||||
}
|
||||
|
||||
for (var i = 0; i < sourceQueries.Length; i++)
|
||||
{
|
||||
var (x, y) = sourceQueries[i];
|
||||
var lookup = cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
Assert.Equal(CacheHitKind.Miss_NotBuilt, lookup.HitKind);
|
||||
Assert.Equal(expected[i].WalkMask, lookup.WalkMask);
|
||||
Assert.Equal(expected[i].WetMask, lookup.WetMask);
|
||||
Assert.Equal(expected[i].WalkZ_N, lookup.WalkZ_N);
|
||||
Assert.Equal(expected[i].WalkZ_NW, lookup.WalkZ_NW);
|
||||
}
|
||||
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal);
|
||||
Assert.Equal(3, cache.GetStats().ResidentChunks);
|
||||
}
|
||||
finally
|
||||
{
|
||||
cache.Clear(); // Releases the FileStream so the delete below succeeds on Windows.
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_MissingFile_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-missing-{System.Guid.NewGuid():N}.swb");
|
||||
Assert.False(cache.TryOpenLazyReader(path, mapId: 1));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_BadMagic_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-badmagic-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
File.WriteAllBytes(path, new byte[]
|
||||
{
|
||||
0xDE, 0xAD, 0xBE, 0xEF,
|
||||
0x01, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00
|
||||
});
|
||||
Assert.False(cache.TryOpenLazyReader(path, mapId: 1));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_TileDataHashMismatch_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
cache.TryGetMask(map, 1500, 1600, sourceZ: 10);
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-stalehash-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
cache.SaveToFile(path, map.MapID);
|
||||
|
||||
// Corrupt the TileDataHash field at byte offset 12 (Magic[4] + Version[4] + MapId[4]).
|
||||
var bytes = File.ReadAllBytes(path);
|
||||
for (var i = 12; i < 20; i++)
|
||||
{
|
||||
bytes[i] ^= 0xFF;
|
||||
}
|
||||
File.WriteAllBytes(path, bytes);
|
||||
|
||||
cache.Clear();
|
||||
Assert.False(cache.TryOpenLazyReader(path, map.MapID));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The lazy reader holds a FileStream open. Saving 100 chunks then opening them all
|
||||
/// must NOT materialize any of them in the resident set — that's the whole point of
|
||||
/// lazy loading on RAM-constrained shards. A query for one specific chunk pulls only
|
||||
/// that one chunk into memory.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LazyReader_DoesNotMaterializeUntilQueried()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
Assert.NotNull(map);
|
||||
|
||||
// Populate a handful of chunks.
|
||||
var coords = new (int, int)[]
|
||||
{
|
||||
(1500, 1600), (1516, 1600), (1500, 1616), (1516, 1616), (1532, 1600)
|
||||
};
|
||||
foreach (var (x, y) in coords)
|
||||
{
|
||||
cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-lazy-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
Assert.Equal(coords.Length, cache.SaveToFile(path, map.MapID));
|
||||
|
||||
cache.Clear();
|
||||
Assert.True(cache.TryOpenLazyReader(path, map.MapID));
|
||||
|
||||
// Open succeeded — but no chunks resident yet.
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
|
||||
// Query one specific chunk: only that chunk lands in the resident set.
|
||||
cache.TryGetMask(map, coords[0].Item1, coords[0].Item2, sourceZ: 10);
|
||||
Assert.Equal(1, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal); // resolved from file, not baker
|
||||
}
|
||||
finally
|
||||
{
|
||||
cache.Clear();
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -67,6 +67,7 @@ public sealed class StepCache
|
|||
{
|
||||
_chunks.Clear();
|
||||
_keysList.Clear();
|
||||
CloseLazyReaders();
|
||||
_hits = 0;
|
||||
_missesNotBuilt = 0;
|
||||
_missesDirtyRebuild = 0;
|
||||
|
|
@ -77,6 +78,111 @@ public sealed class StepCache
|
|||
_buildsTotal = 0;
|
||||
}
|
||||
|
||||
// Per-map open .swb readers, populated by TryOpenLazyReader at startup. Chunks are
|
||||
// fetched on demand from the file when ResolveMissingChunk fires; resident memory
|
||||
// stays bounded by MaxResidentChunks regardless of file size.
|
||||
private readonly Dictionary<int, StepCacheFile.LazyReader> _lazyReaders = new();
|
||||
|
||||
/// <summary>
|
||||
/// Persist all resident chunks for <paramref name="mapId"/> to a .swb file. Returns
|
||||
/// the number of chunks written. The file embeds a TileData fingerprint so a stale
|
||||
/// file (built before a client patch) can be detected and rejected at open time.
|
||||
/// </summary>
|
||||
public int SaveToFile(string path, int mapId)
|
||||
{
|
||||
var matching = 0;
|
||||
foreach (var key in _keysList)
|
||||
{
|
||||
DecodeKey(key, out var keyMapId, out _, out _);
|
||||
if (keyMapId == mapId)
|
||||
{
|
||||
matching++;
|
||||
}
|
||||
}
|
||||
|
||||
var enumerator = _keysList.GetEnumerator();
|
||||
StepCacheFile.Write(path, (uint)mapId, (uint)matching, EmitChunk);
|
||||
enumerator.Dispose();
|
||||
return matching;
|
||||
|
||||
bool EmitChunk(out int chunkX, out int chunkY, out StepChunk chunk)
|
||||
{
|
||||
while (enumerator.MoveNext())
|
||||
{
|
||||
var key = enumerator.Current;
|
||||
DecodeKey(key, out var emittedMapId, out chunkX, out chunkY);
|
||||
if (emittedMapId == mapId)
|
||||
{
|
||||
chunk = _chunks[key];
|
||||
return true;
|
||||
}
|
||||
}
|
||||
chunkX = chunkY = 0;
|
||||
chunk = null!;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Open a .swb file as a lazy backing store for <paramref name="mapId"/>. Reads only
|
||||
/// header + chunk-offset index (~16 bytes per chunk); individual records are fetched
|
||||
/// on demand by <see cref="ResolveMissingChunk"/>. Returns false on missing file,
|
||||
/// magic / version mismatch, or TileData hash mismatch (stale bake).
|
||||
/// </summary>
|
||||
public bool TryOpenLazyReader(string path, int mapId)
|
||||
{
|
||||
var reader = StepCacheFile.OpenForLazy(path);
|
||||
if (reader == null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (reader.MapId != (uint)mapId)
|
||||
{
|
||||
logger.Warning(
|
||||
"StepCache: {Path} declares mapId {FileMapId} but caller requested {RequestedMapId}; ignoring",
|
||||
path, reader.MapId, mapId
|
||||
);
|
||||
reader.Dispose();
|
||||
return false;
|
||||
}
|
||||
|
||||
if (_lazyReaders.TryGetValue(mapId, out var existing))
|
||||
{
|
||||
existing.Dispose();
|
||||
}
|
||||
_lazyReaders[mapId] = reader;
|
||||
|
||||
logger.Information(
|
||||
"StepCache: opened {Path} ({ChunkCount} chunks indexed) for map {MapId}",
|
||||
path, reader.IndexedChunkCount, mapId
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Number of .swb readers currently open. Mostly for tests / telemetry.
|
||||
/// </summary>
|
||||
public int OpenLazyReaderCount => _lazyReaders.Count;
|
||||
|
||||
/// <summary>Test-only diagnostic: does the lazy reader for <paramref name="mapId"/> hold an offset for (chunkX, chunkY)?</summary>
|
||||
internal bool LazyReaderHasChunk(int mapId, int chunkX, int chunkY) =>
|
||||
_lazyReaders.TryGetValue(mapId, out var r) && r.Has(chunkX, chunkY);
|
||||
|
||||
/// <summary>
|
||||
/// Closes all open lazy readers, releasing their underlying file streams. Called from
|
||||
/// <see cref="Clear"/> so test cleanup can delete .swb files (they're held with
|
||||
/// FileShare.Read | FileShare.Delete, so this is mostly belt-and-suspenders).
|
||||
/// </summary>
|
||||
public void CloseLazyReaders()
|
||||
{
|
||||
foreach (var reader in _lazyReaders.Values)
|
||||
{
|
||||
reader.Dispose();
|
||||
}
|
||||
_lazyReaders.Clear();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Probabilistic LRU sample size — picks SampleSize random resident chunks per
|
||||
/// eviction and evicts the oldest of that sample. Approximates true LRU at a tiny
|
||||
|
|
@ -230,10 +336,29 @@ public sealed class StepCache
|
|||
}
|
||||
|
||||
/// <summary>
|
||||
/// Chunk-miss resolution: build the chunk via the runtime baker.
|
||||
/// Chunk-miss resolution: try the lazy file reader for this map first; if there's no
|
||||
/// file or no record at this (chunkX, chunkY), fall back to the runtime baker. The
|
||||
/// file path validates each loaded chunk's MultisVersion against the live sector — a
|
||||
/// stale snapshot triggers a rebuild rather than serving a wrong answer.
|
||||
/// </summary>
|
||||
private StepChunk ResolveMissingChunk(Map map, int chunkX, int chunkY) =>
|
||||
BuildChunk(map, chunkX, chunkY);
|
||||
private StepChunk ResolveMissingChunk(Map map, int chunkX, int chunkY)
|
||||
{
|
||||
if (_lazyReaders.TryGetValue(map.MapID, out var reader))
|
||||
{
|
||||
var loaded = reader.TryReadChunk(chunkX, chunkY);
|
||||
if (loaded != null)
|
||||
{
|
||||
var sector = map.GetRealSector(chunkX, chunkY);
|
||||
if (loaded.BuiltMultisVersion == sector.MultisVersion)
|
||||
{
|
||||
return loaded;
|
||||
}
|
||||
// Snapshot is stale (multis added/removed since the bake). Fall through
|
||||
// to the runtime baker; a future SaveToFile will overwrite the entry.
|
||||
}
|
||||
}
|
||||
return BuildChunk(map, chunkX, chunkY);
|
||||
}
|
||||
|
||||
private StepChunk BuildChunk(Map map, int chunkX, int chunkY)
|
||||
{
|
||||
|
|
|
|||
408
Projects/UOContent/Engines/Pathing/Cache/StepCacheFile.cs
Normal file
408
Projects/UOContent/Engines/Pathing/Cache/StepCacheFile.cs
Normal file
|
|
@ -0,0 +1,408 @@
|
|||
using System;
|
||||
using System.Buffers.Binary;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace Server.Engines.Pathing.Cache;
|
||||
|
||||
/// <summary>
|
||||
/// Binary serializer + lazy reader for the step cache. Persists chunk records to disk
|
||||
/// so a server warm-starts without paying chunk-build cost on the first pathfind through
|
||||
/// a region. Lazy: opening a file reads only the header + chunk-offset index (~few KB
|
||||
/// for tens of thousands of chunks), then individual chunks are seeked + deserialized
|
||||
/// only when the cache asks for them. RAM stays bounded by MaxResidentChunks regardless
|
||||
/// of file size.
|
||||
///
|
||||
/// File layout (little-endian, BufferWriter / BufferReader convention):
|
||||
///
|
||||
/// Header (48 bytes):
|
||||
/// u32 Magic = 0x42575300 ('SWB\0')
|
||||
/// u32 Version = current FormatVersion
|
||||
/// u32 MapId
|
||||
/// u64 TileDataHash XxHash3 over LandTable + ItemTable flags (via
|
||||
/// HashUtility); rejects a load when client tile data has
|
||||
/// shifted under us.
|
||||
/// u64 BakeTimestamp DateTime.UtcNow.Ticks at write time (informational).
|
||||
/// u32 ChunkCount
|
||||
/// u64 IndexOffset File position where the chunk index begins.
|
||||
///
|
||||
/// Per chunk (ChunkCount times, variable size):
|
||||
/// u16 ChunkX
|
||||
/// u16 ChunkY
|
||||
/// u32 BuiltMultisVersion
|
||||
/// u8 HasMultiZ 0 = no MultiZCells follow; 1 = 32 bytes of MultiZCells follow
|
||||
/// byte WalkMask[256]
|
||||
/// byte WetMask[256]
|
||||
/// sbyte SourceZ[256]
|
||||
/// sbyte WalkZN[256]..WalkZNW[256] (8 arrays in N,NE,E,SE,S,SW,W,NW order)
|
||||
/// sbyte SwimZN[256]..SwimZNW[256] (8 arrays in same order)
|
||||
/// [byte MultiZCells[32] — only when HasMultiZ == 1]
|
||||
///
|
||||
/// Index trailer (16 × ChunkCount bytes):
|
||||
/// For each chunk: { u64 chunkKey, u64 fileOffset }
|
||||
///
|
||||
/// Per-chunk size: ~5,393 bytes (no multi-Z) or ~5,425 bytes (with multi-Z).
|
||||
/// LRU bookkeeping (LastTouchedTicks) is intentionally not persisted.
|
||||
/// </summary>
|
||||
internal static class StepCacheFile
|
||||
{
|
||||
public const uint Magic = 0x42575300; // 'SWB\0'
|
||||
public const uint FormatVersion = 1;
|
||||
|
||||
private const int HeaderSize =
|
||||
sizeof(uint) // Magic
|
||||
+ sizeof(uint) // Version
|
||||
+ sizeof(uint) // MapId
|
||||
+ sizeof(ulong) // TileDataHash
|
||||
+ sizeof(ulong) // BakeTimestamp
|
||||
+ sizeof(uint) // ChunkCount
|
||||
+ sizeof(ulong); // IndexOffset
|
||||
|
||||
private const int IndexEntryBytes = sizeof(ulong) + sizeof(ulong); // chunkKey + offset
|
||||
|
||||
private const int BytesPerChunkBase =
|
||||
sizeof(ushort) + sizeof(ushort) + sizeof(uint) + sizeof(byte)
|
||||
+ StepChunk.CellsPerChunk // WalkMask
|
||||
+ StepChunk.CellsPerChunk // WetMask
|
||||
+ StepChunk.CellsPerChunk // SourceZ
|
||||
+ 8 * StepChunk.CellsPerChunk // WalkZ[8]
|
||||
+ 8 * StepChunk.CellsPerChunk; // SwimZ[8]
|
||||
|
||||
private const int BytesPerMultiZ = 32;
|
||||
|
||||
/// <summary>
|
||||
/// Byte offset of the IndexOffset u64 within the header
|
||||
/// (Magic+Version+MapId+TileDataHash+BakeTimestamp+ChunkCount = 32). Patched after chunks land.
|
||||
/// </summary>
|
||||
private const int IndexOffsetFieldPosition = 32;
|
||||
|
||||
public delegate bool ChunkEnumerator(out int chunkX, out int chunkY, out StepChunk chunk);
|
||||
|
||||
/// <summary>
|
||||
/// Computes a stable hash of the loaded TileData flags via XxHash3 (HashUtility).
|
||||
/// Bake files carry this hash so a load can refuse to populate the cache when tile
|
||||
/// data has shifted (client patch, mismatched version) — mismatched data would
|
||||
/// silently skew walkability answers. Hash is stable as long as HashUtility's seed
|
||||
/// constant doesn't change.
|
||||
/// </summary>
|
||||
public static ulong ComputeTileDataHash()
|
||||
{
|
||||
var landTable = TileData.LandTable;
|
||||
var itemTable = TileData.ItemTable;
|
||||
|
||||
// Project just the Flags ulong from each entry into a contiguous byte buffer.
|
||||
// The struct itself contains a string Name (reference) whose object identity isn't
|
||||
// stable across runs, so we can't MemoryMarshal.Cast the whole struct.
|
||||
var bytes = new byte[(landTable.Length + itemTable.Length) * sizeof(ulong)];
|
||||
var span = bytes.AsSpan();
|
||||
|
||||
for (var i = 0; i < landTable.Length; i++)
|
||||
{
|
||||
BinaryPrimitives.WriteUInt64LittleEndian(span[(i * 8)..], (ulong)landTable[i].Flags);
|
||||
}
|
||||
var itemOffset = landTable.Length * 8;
|
||||
for (var i = 0; i < itemTable.Length; i++)
|
||||
{
|
||||
BinaryPrimitives.WriteUInt64LittleEndian(span[(itemOffset + i * 8)..], (ulong)itemTable[i].Flags);
|
||||
}
|
||||
|
||||
return HashUtility.ComputeHash64(bytes);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes the file: header (with placeholder IndexOffset) → chunks (offsets recorded)
|
||||
/// → index trailer → patches the header IndexOffset. <paramref name="chunkCount"/> must
|
||||
/// equal the actual number of chunks <paramref name="next"/> will yield.
|
||||
/// </summary>
|
||||
public static void Write(string path, uint mapId, uint chunkCount, ChunkEnumerator next)
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(path) ?? ".");
|
||||
|
||||
var capacity = HeaderSize
|
||||
+ (BytesPerChunkBase + BytesPerMultiZ) * (int)chunkCount
|
||||
+ IndexEntryBytes * (int)chunkCount;
|
||||
var buffer = new byte[capacity];
|
||||
var w = new BufferWriter(buffer, prefixStr: false);
|
||||
|
||||
w.Write(Magic);
|
||||
w.Write(FormatVersion);
|
||||
w.Write(mapId);
|
||||
w.Write(ComputeTileDataHash());
|
||||
w.Write((ulong)DateTime.UtcNow.Ticks);
|
||||
w.Write(chunkCount);
|
||||
w.Write(0UL); // IndexOffset placeholder, patched after chunks
|
||||
|
||||
var indexEntries = new (ulong key, ulong offset)[chunkCount];
|
||||
var written = 0u;
|
||||
while (next(out var chunkX, out var chunkY, out var chunk))
|
||||
{
|
||||
if (written >= chunkCount)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"StepCacheFile.Write: enumerator yielded more than the declared {chunkCount} chunks"
|
||||
);
|
||||
}
|
||||
var chunkOffset = (ulong)w.Position;
|
||||
WriteChunk(w, chunkX, chunkY, chunk);
|
||||
indexEntries[written] = (PackChunkKey(chunkX, chunkY), chunkOffset);
|
||||
written++;
|
||||
}
|
||||
|
||||
if (written != chunkCount)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"StepCacheFile.Write: declared {chunkCount} chunks but enumerator yielded {written}"
|
||||
);
|
||||
}
|
||||
|
||||
var indexOffset = (ulong)w.Position;
|
||||
for (var i = 0u; i < chunkCount; i++)
|
||||
{
|
||||
w.Write(indexEntries[i].key);
|
||||
w.Write(indexEntries[i].offset);
|
||||
}
|
||||
|
||||
// Patch IndexOffset directly into the buffer (BufferWriter has no Seek).
|
||||
BinaryPrimitives.WriteUInt64LittleEndian(buffer.AsSpan(IndexOffsetFieldPosition, 8), indexOffset);
|
||||
|
||||
var totalBytes = (int)w.Position;
|
||||
File.WriteAllBytes(path, buffer.AsSpan(0, totalBytes).ToArray());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens a .swb file and reads only its header + chunk-offset index. Returns null on
|
||||
/// missing file, magic / version mismatch, or TileDataHash mismatch (a stale bake
|
||||
/// against a freshly patched client). Callers own disposal of the returned reader.
|
||||
/// </summary>
|
||||
public static LazyReader OpenForLazy(string path)
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
FileStream stream = null;
|
||||
try
|
||||
{
|
||||
stream = new FileStream(
|
||||
path,
|
||||
FileMode.Open,
|
||||
FileAccess.Read,
|
||||
FileShare.Read | FileShare.Delete
|
||||
);
|
||||
|
||||
Span<byte> headerBuf = stackalloc byte[HeaderSize];
|
||||
if (stream.Read(headerBuf) != HeaderSize)
|
||||
{
|
||||
stream.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
var magic = BinaryPrimitives.ReadUInt32LittleEndian(headerBuf);
|
||||
if (magic != Magic)
|
||||
{
|
||||
stream.Dispose();
|
||||
return null;
|
||||
}
|
||||
var version = BinaryPrimitives.ReadUInt32LittleEndian(headerBuf[4..]);
|
||||
if (version != FormatVersion)
|
||||
{
|
||||
stream.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
var mapId = BinaryPrimitives.ReadUInt32LittleEndian(headerBuf[8..]);
|
||||
var tileDataHash = BinaryPrimitives.ReadUInt64LittleEndian(headerBuf[12..]);
|
||||
var bakeTimestamp = BinaryPrimitives.ReadUInt64LittleEndian(headerBuf[20..]);
|
||||
var chunkCount = BinaryPrimitives.ReadUInt32LittleEndian(headerBuf[28..]);
|
||||
var indexOffset = BinaryPrimitives.ReadUInt64LittleEndian(headerBuf[32..]);
|
||||
|
||||
if (tileDataHash != ComputeTileDataHash())
|
||||
{
|
||||
stream.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
// Read the chunk-offset index in one shot.
|
||||
var indexBytes = (int)chunkCount * IndexEntryBytes;
|
||||
var indexBuf = new byte[indexBytes];
|
||||
stream.Position = (long)indexOffset;
|
||||
if (stream.Read(indexBuf, 0, indexBytes) != indexBytes)
|
||||
{
|
||||
stream.Dispose();
|
||||
return null;
|
||||
}
|
||||
|
||||
var offsets = new Dictionary<ulong, ulong>((int)chunkCount);
|
||||
for (var i = 0; i < chunkCount; i++)
|
||||
{
|
||||
var entry = indexBuf.AsSpan(i * IndexEntryBytes);
|
||||
var key = BinaryPrimitives.ReadUInt64LittleEndian(entry);
|
||||
var off = BinaryPrimitives.ReadUInt64LittleEndian(entry[8..]);
|
||||
offsets[key] = off;
|
||||
}
|
||||
|
||||
return new LazyReader(stream, mapId, tileDataHash, bakeTimestamp, chunkCount, offsets);
|
||||
}
|
||||
catch
|
||||
{
|
||||
stream?.Dispose();
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static ulong PackChunkKey(int chunkX, int chunkY) => ((ulong)(uint)chunkX << 32) | (uint)chunkY;
|
||||
|
||||
private static void WriteChunk(BufferWriter w, int chunkX, int chunkY, StepChunk chunk)
|
||||
{
|
||||
w.Write((ushort)chunkX);
|
||||
w.Write((ushort)chunkY);
|
||||
w.Write((uint)chunk.BuiltMultisVersion);
|
||||
|
||||
var multiZ = chunk.GetMultiZCellsForSerialization();
|
||||
w.Write((byte)(multiZ != null ? 1 : 0));
|
||||
|
||||
w.Write(chunk.WalkMask);
|
||||
w.Write(chunk.WetMask);
|
||||
WriteSBytes(w, chunk.SourceZ);
|
||||
|
||||
WriteSBytes(w, chunk.WalkZN);
|
||||
WriteSBytes(w, chunk.WalkZNE);
|
||||
WriteSBytes(w, chunk.WalkZE);
|
||||
WriteSBytes(w, chunk.WalkZSE);
|
||||
WriteSBytes(w, chunk.WalkZS);
|
||||
WriteSBytes(w, chunk.WalkZSW);
|
||||
WriteSBytes(w, chunk.WalkZW);
|
||||
WriteSBytes(w, chunk.WalkZNW);
|
||||
|
||||
WriteSBytes(w, chunk.SwimZN);
|
||||
WriteSBytes(w, chunk.SwimZNE);
|
||||
WriteSBytes(w, chunk.SwimZE);
|
||||
WriteSBytes(w, chunk.SwimZSE);
|
||||
WriteSBytes(w, chunk.SwimZS);
|
||||
WriteSBytes(w, chunk.SwimZSW);
|
||||
WriteSBytes(w, chunk.SwimZW);
|
||||
WriteSBytes(w, chunk.SwimZNW);
|
||||
|
||||
if (multiZ != null)
|
||||
{
|
||||
w.Write(multiZ);
|
||||
}
|
||||
}
|
||||
|
||||
private static StepChunk ReadChunk(byte[] buffer)
|
||||
{
|
||||
var r = new BufferReader(buffer);
|
||||
// Skip ChunkX + ChunkY (already known via the index lookup).
|
||||
r.ReadUShort();
|
||||
r.ReadUShort();
|
||||
var multisVersion = (int)r.ReadUInt();
|
||||
var hasMultiZ = r.ReadByte() != 0;
|
||||
|
||||
var chunk = new StepChunk { BuiltMultisVersion = multisVersion };
|
||||
|
||||
r.Read(chunk.WalkMask);
|
||||
r.Read(chunk.WetMask);
|
||||
ReadSBytes(r, chunk.SourceZ);
|
||||
|
||||
ReadSBytes(r, chunk.WalkZN);
|
||||
ReadSBytes(r, chunk.WalkZNE);
|
||||
ReadSBytes(r, chunk.WalkZE);
|
||||
ReadSBytes(r, chunk.WalkZSE);
|
||||
ReadSBytes(r, chunk.WalkZS);
|
||||
ReadSBytes(r, chunk.WalkZSW);
|
||||
ReadSBytes(r, chunk.WalkZW);
|
||||
ReadSBytes(r, chunk.WalkZNW);
|
||||
|
||||
ReadSBytes(r, chunk.SwimZN);
|
||||
ReadSBytes(r, chunk.SwimZNE);
|
||||
ReadSBytes(r, chunk.SwimZE);
|
||||
ReadSBytes(r, chunk.SwimZSE);
|
||||
ReadSBytes(r, chunk.SwimZS);
|
||||
ReadSBytes(r, chunk.SwimZSW);
|
||||
ReadSBytes(r, chunk.SwimZW);
|
||||
ReadSBytes(r, chunk.SwimZNW);
|
||||
|
||||
if (hasMultiZ)
|
||||
{
|
||||
var multiZ = new byte[BytesPerMultiZ];
|
||||
r.Read(multiZ);
|
||||
chunk.RestoreMultiZCellsFromSerialization(multiZ);
|
||||
}
|
||||
|
||||
return chunk;
|
||||
}
|
||||
|
||||
private static void WriteSBytes(BufferWriter w, sbyte[] arr) =>
|
||||
w.Write(MemoryMarshal.Cast<sbyte, byte>(arr.AsSpan()));
|
||||
|
||||
private static void ReadSBytes(BufferReader r, sbyte[] arr) =>
|
||||
r.Read(MemoryMarshal.Cast<sbyte, byte>(arr.AsSpan()));
|
||||
|
||||
/// <summary>
|
||||
/// Open handle on a .swb file. Holds the FileStream + chunk-offset index. Chunks are
|
||||
/// fetched on demand via <see cref="TryReadChunk"/>; only the records actually queried
|
||||
/// are ever materialized. Dispose releases the underlying stream.
|
||||
/// </summary>
|
||||
internal sealed class LazyReader : IDisposable
|
||||
{
|
||||
private FileStream _stream;
|
||||
private readonly Dictionary<ulong, ulong> _offsets;
|
||||
private byte[] _buffer;
|
||||
|
||||
public uint MapId { get; }
|
||||
public ulong TileDataHash { get; }
|
||||
public ulong BakeTimestamp { get; }
|
||||
public uint ChunkCount { get; }
|
||||
public int IndexedChunkCount => _offsets.Count;
|
||||
|
||||
public bool Has(int chunkX, int chunkY) => _offsets.ContainsKey(PackChunkKey(chunkX, chunkY));
|
||||
|
||||
internal LazyReader(
|
||||
FileStream stream, uint mapId, ulong tileDataHash, ulong bakeTimestamp,
|
||||
uint chunkCount, Dictionary<ulong, ulong> offsets
|
||||
)
|
||||
{
|
||||
_stream = stream;
|
||||
MapId = mapId;
|
||||
TileDataHash = tileDataHash;
|
||||
BakeTimestamp = bakeTimestamp;
|
||||
ChunkCount = chunkCount;
|
||||
_offsets = offsets;
|
||||
_buffer = new byte[BytesPerChunkBase + BytesPerMultiZ];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the chunk record at (<paramref name="chunkX"/>, <paramref name="chunkY"/>)
|
||||
/// from the file, or null if the file doesn't contain it. Single seek + bulk read;
|
||||
/// no allocations beyond the returned StepChunk and its arrays.
|
||||
/// </summary>
|
||||
public StepChunk TryReadChunk(int chunkX, int chunkY)
|
||||
{
|
||||
if (_stream == null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var key = PackChunkKey(chunkX, chunkY);
|
||||
if (!_offsets.TryGetValue(key, out var offset))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
_stream.Position = (long)offset;
|
||||
// Try to read the maximum size; the file may have less remaining, which is OK
|
||||
// since BufferReader stops at the bytes it actually needs.
|
||||
var read = _stream.Read(_buffer, 0, _buffer.Length);
|
||||
return read < BytesPerChunkBase ? null : ReadChunk(_buffer);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_stream?.Dispose();
|
||||
_stream = null;
|
||||
_buffer = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -55,4 +55,16 @@ internal sealed class StepChunk
|
|||
_multiZCells ??= new byte[32];
|
||||
_multiZCells[cellIndex >> 3] |= (byte)(1 << (cellIndex & 7));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Serialization hook for <see cref="StepCacheFile"/>: returns the multi-Z bitmap,
|
||||
/// or null if no cells in this chunk are multi-Z. Read-only — callers must not mutate.
|
||||
/// </summary>
|
||||
internal byte[] GetMultiZCellsForSerialization() => _multiZCells;
|
||||
|
||||
/// <summary>
|
||||
/// Deserialization hook: assigns the multi-Z bitmap from a file load. Caller is
|
||||
/// responsible for passing a 32-byte array (or null for "no cells multi-Z").
|
||||
/// </summary>
|
||||
internal void RestoreMultiZCellsFromSerialization(byte[] multiZ) => _multiZCells = multiZ;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
using System.IO;
|
||||
using Server.Engines.Pathing.Cache;
|
||||
|
||||
namespace Server.Engines.Pathing;
|
||||
|
|
@ -5,14 +6,49 @@ namespace Server.Engines.Pathing;
|
|||
/// <summary>
|
||||
/// Admin commands for inspecting and operating the pathfinding step cache.
|
||||
/// [PathCacheStats — current resident-chunk count + hit/miss/eviction telemetry.
|
||||
/// [PathCacheClear — drop all cached chunks and zero counters.
|
||||
/// [PathCacheClear — drop all cached chunks, close lazy readers, zero counters.
|
||||
/// [PathCacheSave — persist resident chunks per map to Data/Pathfinding/<mapId>.swb.
|
||||
/// [PathCacheLoad — open those files as lazy backing stores. Also runs at startup.
|
||||
/// </summary>
|
||||
public static class PathCacheCommands
|
||||
{
|
||||
private static string PathFor(int mapId) =>
|
||||
Path.Combine(Core.BaseDirectory, "Data", "Pathfinding", $"{mapId}.swb");
|
||||
|
||||
public static void Configure()
|
||||
{
|
||||
// Resident-chunk cap is shard-tunable. Default 8192 ≈ 40 MB; small shards may
|
||||
// want lower, large shards (or full-map bakes) may want higher. Setting is
|
||||
// written back to server.cfg on first boot for discoverability.
|
||||
StepCache.Instance.MaxResidentChunks = ServerConfiguration.GetOrUpdateSetting(
|
||||
"pathfinding.maxResidentChunks",
|
||||
8192
|
||||
);
|
||||
|
||||
CommandSystem.Register("PathCacheStats", AccessLevel.Administrator, OnPathCacheStats);
|
||||
CommandSystem.Register("PathCacheClear", AccessLevel.Administrator, OnPathCacheClear);
|
||||
CommandSystem.Register("PathCacheSave", AccessLevel.Administrator, OnPathCacheSave);
|
||||
CommandSystem.Register("PathCacheLoad", AccessLevel.Administrator, OnPathCacheLoad);
|
||||
AutoLoadAtStartup();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Open Data/Pathfinding/<mapId>.swb as a lazy backing store for every map.
|
||||
/// Reads only the header + chunk-offset index up front (~16 bytes per chunk);
|
||||
/// individual chunk records are fetched on demand when the cache asks for them.
|
||||
/// RAM stays bounded by MaxResidentChunks regardless of file size.
|
||||
/// </summary>
|
||||
private static void AutoLoadAtStartup()
|
||||
{
|
||||
for (var i = 0; i < Map.Maps.Length; i++)
|
||||
{
|
||||
var map = Map.Maps[i];
|
||||
if (map == null || map == Map.Internal)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID);
|
||||
}
|
||||
}
|
||||
|
||||
[Usage("PathCacheStats")]
|
||||
|
|
@ -37,4 +73,51 @@ public static class PathCacheCommands
|
|||
StepCache.Instance.Clear();
|
||||
e.Mobile.SendMessage($"StepCache cleared: {residentBefore} chunks dropped, counters reset.");
|
||||
}
|
||||
|
||||
[Usage("PathCacheSave")]
|
||||
[Description("Persists resident StepCache chunks for every loaded map to Data/Pathfinding/<mapId>.swb.")]
|
||||
private static void OnPathCacheSave(CommandEventArgs e)
|
||||
{
|
||||
var totalChunks = 0;
|
||||
var totalMaps = 0;
|
||||
for (var i = 0; i < Map.Maps.Length; i++)
|
||||
{
|
||||
var map = Map.Maps[i];
|
||||
if (map == null || map == Map.Internal)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
var path = PathFor(map.MapID);
|
||||
var written = StepCache.Instance.SaveToFile(path, map.MapID);
|
||||
if (written > 0)
|
||||
{
|
||||
totalChunks += written;
|
||||
totalMaps++;
|
||||
e.Mobile.SendMessage($" map {map.MapID}: {written} chunks → {path}");
|
||||
}
|
||||
}
|
||||
e.Mobile.SendMessage($"StepCache saved: {totalChunks} chunks across {totalMaps} map(s).");
|
||||
}
|
||||
|
||||
[Usage("PathCacheLoad")]
|
||||
[Description("Opens Data/Pathfinding/<mapId>.swb as a lazy backing store for every map. Chunks are fetched on demand, so RAM stays bounded by the LRU cap regardless of file size.")]
|
||||
private static void OnPathCacheLoad(CommandEventArgs e)
|
||||
{
|
||||
var openedMaps = 0;
|
||||
for (var i = 0; i < Map.Maps.Length; i++)
|
||||
{
|
||||
var map = Map.Maps[i];
|
||||
if (map == null || map == Map.Internal)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
if (StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID))
|
||||
{
|
||||
openedMaps++;
|
||||
}
|
||||
}
|
||||
e.Mobile.SendMessage(
|
||||
$"StepCache: opened {openedMaps} map(s) for lazy loading (total readers open: {StepCache.Instance.OpenLazyReaderCount})."
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue