feat(pathfinding): lazy .swb backing store for the step cache (#2448)
## Summary Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them. The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all. Builds on PR #2447. ## What changed - **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary. - **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them. - **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity. - **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map. - **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime. - **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector). - **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly. - **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map. - **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow. - **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings. ## File layout (v1) ``` Header (48 bytes): u32 Magic = 0x42575300 ('SWB\0') u32 Version = 1 u32 MapId u64 TileDataHash XxHash3 over LandTable + ItemTable flags (HashUtility) u64 BakeTimestamp informational u32 ChunkCount u64 IndexOffset file position where the chunk index begins Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z): u16 ChunkX u16 ChunkY u32 BuiltMultisVersion u8 HasMultiZ byte WalkMask[256], WetMask[256] sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256] [byte MultiZCells[32] when HasMultiZ == 1] Index trailer (16 × ChunkCount bytes): (u64 chunkKey, u64 fileOffset) ``` ## Memory math | Scenario | Disk file | RAM at boot | Notes | |---|---|---|---| | Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. | | Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. | | Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. | | All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. | ## Hash choice (FNV-1a → XxHash3) The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`: - ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere. - Stronger collision resistance and distribution. - Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern. - Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
This commit is contained in:
Kamron Batman
GitHub
parent
9066e8fd00
commit
7c9215d97c
6 changed files with 861 additions and 4 deletions
|
|
@ -0,0 +1,213 @@
|
|||
using System.IO;
|
||||
using Server.Engines.Pathing.Cache;
|
||||
using Xunit;
|
||||
|
||||
namespace Server.Tests.Pathfinding;
|
||||
|
||||
[Collection("Sequential Pathfinding Tests")]
|
||||
public class StepCacheFileTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Round-trip a populated cache through a .swb file under lazy loading: build chunks,
|
||||
/// save, clear (closes lazy readers + drops residents), open as lazy backing store,
|
||||
/// then re-query and verify chunks come back from the file (not the runtime baker).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void RoundTrip_LazyLoad_PreservesChunkMaskAndZ()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
Assert.NotNull(map);
|
||||
|
||||
// Populate three distinct chunks by querying different sectors.
|
||||
var sourceQueries = new[] { (1500, 1600), (1516, 1600), (1500, 1616) };
|
||||
foreach (var (x, y) in sourceQueries)
|
||||
{
|
||||
cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
Assert.Equal(3, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(3L, cache.GetStats().BuildsTotal);
|
||||
|
||||
// Snapshot the answers we expect to recover after round-trip.
|
||||
var expected = new StepMask[sourceQueries.Length];
|
||||
for (var i = 0; i < sourceQueries.Length; i++)
|
||||
{
|
||||
var (x, y) = sourceQueries[i];
|
||||
expected[i] = cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-roundtrip-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
var written = cache.SaveToFile(path, map.MapID);
|
||||
Assert.Equal(3, written);
|
||||
|
||||
cache.Clear();
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
|
||||
Assert.True(cache.TryOpenLazyReader(path, map.MapID));
|
||||
Assert.Equal(1, cache.OpenLazyReaderCount);
|
||||
|
||||
// Lazy: opening doesn't materialize chunks, so the resident set stays empty
|
||||
// until we query.
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal);
|
||||
|
||||
// Sanity: the lazy reader's index covers each chunk we're about to query.
|
||||
foreach (var (x, y) in sourceQueries)
|
||||
{
|
||||
Assert.True(cache.LazyReaderHasChunk(map.MapID, x >> 4, y >> 4),
|
||||
$"lazy reader missing chunk ({x >> 4},{y >> 4})");
|
||||
}
|
||||
|
||||
for (var i = 0; i < sourceQueries.Length; i++)
|
||||
{
|
||||
var (x, y) = sourceQueries[i];
|
||||
var lookup = cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
Assert.Equal(CacheHitKind.Miss_NotBuilt, lookup.HitKind);
|
||||
Assert.Equal(expected[i].WalkMask, lookup.WalkMask);
|
||||
Assert.Equal(expected[i].WetMask, lookup.WetMask);
|
||||
Assert.Equal(expected[i].WalkZ_N, lookup.WalkZ_N);
|
||||
Assert.Equal(expected[i].WalkZ_NW, lookup.WalkZ_NW);
|
||||
}
|
||||
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal);
|
||||
Assert.Equal(3, cache.GetStats().ResidentChunks);
|
||||
}
|
||||
finally
|
||||
{
|
||||
cache.Clear(); // Releases the FileStream so the delete below succeeds on Windows.
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_MissingFile_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-missing-{System.Guid.NewGuid():N}.swb");
|
||||
Assert.False(cache.TryOpenLazyReader(path, mapId: 1));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_BadMagic_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-badmagic-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
File.WriteAllBytes(path, new byte[]
|
||||
{
|
||||
0xDE, 0xAD, 0xBE, 0xEF,
|
||||
0x01, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00
|
||||
});
|
||||
Assert.False(cache.TryOpenLazyReader(path, mapId: 1));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TryOpenLazyReader_TileDataHashMismatch_ReturnsFalse()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
cache.TryGetMask(map, 1500, 1600, sourceZ: 10);
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-stalehash-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
cache.SaveToFile(path, map.MapID);
|
||||
|
||||
// Corrupt the TileDataHash field at byte offset 12 (Magic[4] + Version[4] + MapId[4]).
|
||||
var bytes = File.ReadAllBytes(path);
|
||||
for (var i = 12; i < 20; i++)
|
||||
{
|
||||
bytes[i] ^= 0xFF;
|
||||
}
|
||||
File.WriteAllBytes(path, bytes);
|
||||
|
||||
cache.Clear();
|
||||
Assert.False(cache.TryOpenLazyReader(path, map.MapID));
|
||||
Assert.Equal(0, cache.OpenLazyReaderCount);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The lazy reader holds a FileStream open. Saving 100 chunks then opening them all
|
||||
/// must NOT materialize any of them in the resident set — that's the whole point of
|
||||
/// lazy loading on RAM-constrained shards. A query for one specific chunk pulls only
|
||||
/// that one chunk into memory.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LazyReader_DoesNotMaterializeUntilQueried()
|
||||
{
|
||||
var cache = StepCache.Instance;
|
||||
cache.Clear();
|
||||
|
||||
var map = Map.Maps[1];
|
||||
Assert.NotNull(map);
|
||||
|
||||
// Populate a handful of chunks.
|
||||
var coords = new (int, int)[]
|
||||
{
|
||||
(1500, 1600), (1516, 1600), (1500, 1616), (1516, 1616), (1532, 1600)
|
||||
};
|
||||
foreach (var (x, y) in coords)
|
||||
{
|
||||
cache.TryGetMask(map, x, y, sourceZ: 10);
|
||||
}
|
||||
|
||||
var path = Path.Combine(Path.GetTempPath(), $"step-cache-lazy-{System.Guid.NewGuid():N}.swb");
|
||||
try
|
||||
{
|
||||
Assert.Equal(coords.Length, cache.SaveToFile(path, map.MapID));
|
||||
|
||||
cache.Clear();
|
||||
Assert.True(cache.TryOpenLazyReader(path, map.MapID));
|
||||
|
||||
// Open succeeded — but no chunks resident yet.
|
||||
Assert.Equal(0, cache.GetStats().ResidentChunks);
|
||||
|
||||
// Query one specific chunk: only that chunk lands in the resident set.
|
||||
cache.TryGetMask(map, coords[0].Item1, coords[0].Item2, sourceZ: 10);
|
||||
Assert.Equal(1, cache.GetStats().ResidentChunks);
|
||||
Assert.Equal(0L, cache.GetStats().BuildsTotal); // resolved from file, not baker
|
||||
}
|
||||
finally
|
||||
{
|
||||
cache.Clear();
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue