## Summary Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them. The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all. Builds on PR #2447. ## What changed - **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary. - **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them. - **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity. - **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map. - **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime. - **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector). - **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly. - **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map. - **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow. - **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings. ## File layout (v1) ``` Header (48 bytes): u32 Magic = 0x42575300 ('SWB\0') u32 Version = 1 u32 MapId u64 TileDataHash XxHash3 over LandTable + ItemTable flags (HashUtility) u64 BakeTimestamp informational u32 ChunkCount u64 IndexOffset file position where the chunk index begins Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z): u16 ChunkX u16 ChunkY u32 BuiltMultisVersion u8 HasMultiZ byte WalkMask[256], WetMask[256] sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256] [byte MultiZCells[32] when HasMultiZ == 1] Index trailer (16 × ChunkCount bytes): (u64 chunkKey, u64 fileOffset) ``` ## Memory math | Scenario | Disk file | RAM at boot | Notes | |---|---|---|---| | Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. | | Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. | | Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. | | All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. | ## Hash choice (FNV-1a → XxHash3) The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`: - ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere. - Stronger collision resistance and distribution. - Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern. - Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
123 lines
5.1 KiB
C#
123 lines
5.1 KiB
C#
using System.IO;
|
|
using Server.Engines.Pathing.Cache;
|
|
|
|
namespace Server.Engines.Pathing;
|
|
|
|
/// <summary>
|
|
/// Admin commands for inspecting and operating the pathfinding step cache.
|
|
/// [PathCacheStats — current resident-chunk count + hit/miss/eviction telemetry.
|
|
/// [PathCacheClear — drop all cached chunks, close lazy readers, zero counters.
|
|
/// [PathCacheSave — persist resident chunks per map to Data/Pathfinding/<mapId>.swb.
|
|
/// [PathCacheLoad — open those files as lazy backing stores. Also runs at startup.
|
|
/// </summary>
|
|
public static class PathCacheCommands
|
|
{
|
|
private static string PathFor(int mapId) =>
|
|
Path.Combine(Core.BaseDirectory, "Data", "Pathfinding", $"{mapId}.swb");
|
|
|
|
public static void Configure()
|
|
{
|
|
// Resident-chunk cap is shard-tunable. Default 8192 ≈ 40 MB; small shards may
|
|
// want lower, large shards (or full-map bakes) may want higher. Setting is
|
|
// written back to server.cfg on first boot for discoverability.
|
|
StepCache.Instance.MaxResidentChunks = ServerConfiguration.GetOrUpdateSetting(
|
|
"pathfinding.maxResidentChunks",
|
|
8192
|
|
);
|
|
|
|
CommandSystem.Register("PathCacheStats", AccessLevel.Administrator, OnPathCacheStats);
|
|
CommandSystem.Register("PathCacheClear", AccessLevel.Administrator, OnPathCacheClear);
|
|
CommandSystem.Register("PathCacheSave", AccessLevel.Administrator, OnPathCacheSave);
|
|
CommandSystem.Register("PathCacheLoad", AccessLevel.Administrator, OnPathCacheLoad);
|
|
AutoLoadAtStartup();
|
|
}
|
|
|
|
/// <summary>
|
|
/// Open Data/Pathfinding/<mapId>.swb as a lazy backing store for every map.
|
|
/// Reads only the header + chunk-offset index up front (~16 bytes per chunk);
|
|
/// individual chunk records are fetched on demand when the cache asks for them.
|
|
/// RAM stays bounded by MaxResidentChunks regardless of file size.
|
|
/// </summary>
|
|
private static void AutoLoadAtStartup()
|
|
{
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID);
|
|
}
|
|
}
|
|
|
|
[Usage("PathCacheStats")]
|
|
[Description("Reports StepCache resident-chunk count and hit/miss/eviction telemetry.")]
|
|
private static void OnPathCacheStats(CommandEventArgs e)
|
|
{
|
|
var stats = StepCache.Instance.GetStats();
|
|
var from = e.Mobile;
|
|
|
|
from.SendMessage($"StepCache: {stats.ResidentChunks} chunks resident");
|
|
from.SendMessage($" builds={stats.BuildsTotal} hits={stats.Hits}");
|
|
from.SendMessage($" miss(notBuilt)={stats.MissesNotBuilt} miss(dirty)={stats.MissesDirtyRebuild}");
|
|
from.SendMessage($" fallthru(multiZ)={stats.FallthroughMultiZ} fallthru(offMap)={stats.FallthroughOffMap} fallthru(srcZ)={stats.FallthroughSourceZMismatch}");
|
|
from.SendMessage($" evictions(lruCap)={stats.EvictionsByLruCap}");
|
|
}
|
|
|
|
[Usage("PathCacheClear")]
|
|
[Description("Drops all StepCache resident chunks and zeros the telemetry counters.")]
|
|
private static void OnPathCacheClear(CommandEventArgs e)
|
|
{
|
|
var residentBefore = StepCache.Instance.GetStats().ResidentChunks;
|
|
StepCache.Instance.Clear();
|
|
e.Mobile.SendMessage($"StepCache cleared: {residentBefore} chunks dropped, counters reset.");
|
|
}
|
|
|
|
[Usage("PathCacheSave")]
|
|
[Description("Persists resident StepCache chunks for every loaded map to Data/Pathfinding/<mapId>.swb.")]
|
|
private static void OnPathCacheSave(CommandEventArgs e)
|
|
{
|
|
var totalChunks = 0;
|
|
var totalMaps = 0;
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
var path = PathFor(map.MapID);
|
|
var written = StepCache.Instance.SaveToFile(path, map.MapID);
|
|
if (written > 0)
|
|
{
|
|
totalChunks += written;
|
|
totalMaps++;
|
|
e.Mobile.SendMessage($" map {map.MapID}: {written} chunks → {path}");
|
|
}
|
|
}
|
|
e.Mobile.SendMessage($"StepCache saved: {totalChunks} chunks across {totalMaps} map(s).");
|
|
}
|
|
|
|
[Usage("PathCacheLoad")]
|
|
[Description("Opens Data/Pathfinding/<mapId>.swb as a lazy backing store for every map. Chunks are fetched on demand, so RAM stays bounded by the LRU cap regardless of file size.")]
|
|
private static void OnPathCacheLoad(CommandEventArgs e)
|
|
{
|
|
var openedMaps = 0;
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
if (StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID))
|
|
{
|
|
openedMaps++;
|
|
}
|
|
}
|
|
e.Mobile.SendMessage(
|
|
$"StepCache: opened {openedMaps} map(s) for lazy loading (total readers open: {StepCache.Instance.OpenLazyReaderCount})."
|
|
);
|
|
}
|
|
}
|