## Summary Closes the Cold-cache regression flagged in PR #2450. `StepCache.TryGetMask` no longer eagerly runs `BuildChunk` on the first miss for a chunk that isn't in a `.swb` lazy reader. Instead it returns `Fallthrough_NotBuilt` and the caller (`BitmapAStarAlgorithm`) takes the per-cell slow path. The chunk is only promoted to the bitmap fast path after the **second** miss within a 30-second window, filtering single-touch pass-throughs. This makes BitmapAStar's worst-case (cold cache + short hops) collapse from **12–47× slower** than FastAStar to **roughly the same**, which is the floor the slow path can deliver. Steady-state warm performance (the actual deliverable) is unchanged from PR-5 — it was always the cache fast path. ## The pet-follow scenario this fixes A mounted player at ~4 tiles/sec with a pet/hireable following will trigger an NPC pathfind every 100–300 ms. Each pathfind is 1–6 tiles. As the player crosses chunk boundaries (~4 sec/chunk), the pet's first pathfind in the new chunk under the previous behavior triggered a full ~700 µs `BuildChunk` for a chunk the player would leave shortly after. At 50–100 mobiles per shard, this exceeded the 8 ms tick budget. PR-5 BDN data showed scenarios 6–9 (2–8 tile NPC perception) at 2,300–3,700 µs Cold vs FastAStar's 80–200 µs. Under the new gate: - First miss → `Fallthrough_NotBuilt` → caller uses slow path (~30–50 µs short path). No `BuildChunk`. No allocation. - Player keeps moving → chunk never gets a second touch within window → never promoted, no rot. - NPC patrolling a fixed territory → repeatedly hits the same chunks → second touch within window → promote → cache fast path on subsequent calls. ## What changed - **`CacheHitKind.Fallthrough_NotBuilt = 6`** + **`CacheStats.FallthroughNotBuilt`** counter. `IsHit=false`, so the caller routes to slow path. - **`StepCache._chunkMissTracker`** — `Dictionary<long, ChunkMissState>` capped at 4096 entries. State is `(byte missCount, uint lastMissTickStamp)` keyed by chunk key. Window-expired entries reset count to 1; capacity overflow prunes window-old entries first. - **`StepCache.MissPromotionThreshold`** (default `2`) and **`StepCache.MissPromotionWindowMs`** (default `30_000`) — tunable, can be wired through `ServerConfiguration` if shards want different policy. Setting threshold to `1` restores legacy eager-build behavior (used by tests that prime chunks via single `TryGetMask` call). - **`StepCache.TryGetMask` miss branch** — try lazy reader first (file-loaded chunks bypass the tracker entirely; an `.swb` represents an explicit prior decision to keep the chunk warm). Otherwise consult the tracker. - **`BitmapAStarAlgorithm.GetSuccessorsSlowPath`** now layers `IsBlockedByDynamic` on top of `CalcMoves.CheckMovement`. Previously the slow path only ran for `CanFly` creatures and rare cache fallthroughs — `CheckMovement` doesn't iterate same-cell mobiles, so the bitmap fast path's `IsBlockedByDynamic` was the only mobile-blocking check. Now first-touch pathfinds run through the slow path, so the gap had to close. ## Tests 50 pathfinding tests pass (was 47). New / updated: - **`TryGetMask_FirstTouchOnUnbuiltChunk_DefersBuildAndReturnsFallthrough`** — single TryGetMask call returns `Fallthrough_NotBuilt`, no chunk built, no allocation. - **`TryGetMask_SecondTouchWithinWindow_PromotesAndBuilds`** — second call inside the 30s window builds + serves. - **`TryGetMask_SecondTouchAfterWindow_RestartsCounterAndDefers`** — second call outside the window restarts the count, returns Fallthrough again. - **`TryGetMask_DistinctChunks_TrackedIndependently`** — counters are per-chunk; one touch on each of two adjacent chunks both stay in fallthrough. - **`LazyReaderHit_BypassesMissTrackerOnFirstTouch`** — open `.swb` + first touch hits without consulting the tracker. Production with `.swb` loaded skips the gate entirely. - **`MultisVersion_Bump_TriggersDirtyRebuild`** — updated to reflect the new 3-step flow (Fallthrough → Miss_NotBuilt → Miss_DirtyRebuild). - Tests that prime chunks via a single `TryGetMask` call (multi-Z, Tier4, lifecycle, parity, BitmapAStar uses-cache) set `MissPromotionThreshold = 1` to opt into eager behavior. ## Expected BDN impact The Cold column from PR-5's BDN should change as follows once the bench's submodule pointer is updated to this branch: | # | Scenario | Cold (PR-5) | Cold (PR-6 expected) | FastAStar Cold | |--:|-----------------|-------------:|---------------------:|---------------:| | 2 | sewer corridor | 1,627 µs | ~36 µs | 36 µs | | 4 | causeway | 1,533 µs | ~39 µs | 39 µs | | 6 | pet 2-tile | 2,364 µs | ~80 µs | 81 µs | | 8 | npc 5-tile | 3,708 µs | ~140 µs | 141 µs | | 9 | npc 8-tile | 2,386 µs | ~200 µs | 197 µs | WarmNoFile and LazyWarm rows should be unchanged — they were always cache-warm. The miss tracker only fires when neither resident chunks nor the lazy reader can satisfy the request. ## Future work (not in this PR) - **Background-thread bake**: builds outside the game thread so even promoted chunks don't pay the 700 µs build cost on the main thread. Rule 10 (no Task.Run) applies, so this needs careful design — the bake is a pure data transform but main-thread synchronization on chunk-state transitions has to be threaded through. Defer to a follow-up. - **Long-traverse BDN scenario**: a multi-Find benchmark simulating 50 pet repaths across chunk transitions. Requires restructuring the bench harness; the existing 10-scenario corpus + Cold provider already exercises the gate. - **Swim sourceZ bake**: scenario 5 (sea serpent) shows 56 B alloc on warm paths because the cache's SourceZ is computed under default-walker rules. Swim creatures fall through to slow path. Independent of this PR.
221 lines
9.6 KiB
C#
221 lines
9.6 KiB
C#
using System.IO;
|
|
using Server.Engines.Pathing.Cache;
|
|
|
|
namespace Server.Engines.Pathing;
|
|
|
|
/// <summary>
|
|
/// Admin commands for inspecting and operating the pathfinding step cache.
|
|
/// [PathCacheStats — current resident-chunk count + hit/miss/eviction telemetry.
|
|
/// [PathCacheClear — drop all cached chunks, close lazy readers, zero counters.
|
|
/// [PathBake — walk a whole map building the full static cache, then save it.
|
|
/// [PathCacheSave — persist resident chunks per map to Data/Pathfinding/<mapId>.swb.
|
|
/// [PathCacheLoad — open those files as lazy backing stores. Also runs at startup.
|
|
/// [PathRecord — toggle JSONL telemetry capture for replay / benchmark corpora.
|
|
///
|
|
/// The step cache works WITHOUT any .swb file — chunks build on demand as creatures path.
|
|
/// A baked .swb is an optional optimization that removes first-pathfind-after-boot latency
|
|
/// for shard owners who want it; <see cref="OnPathBake"/> is how you produce one.
|
|
/// </summary>
|
|
public static class PathCacheCommands
|
|
{
|
|
private static string PathFor(int mapId) =>
|
|
Path.Combine(Core.BaseDirectory, "Data", "Pathfinding", $"{mapId}.swb");
|
|
|
|
public static void Configure()
|
|
{
|
|
// Resident-chunk cap is shard-tunable. Default 8192 ≈ 40 MB; small shards may
|
|
// want lower, large shards (or full-map bakes) may want higher. Setting is
|
|
// written back to server.cfg on first boot for discoverability.
|
|
StepCache.Instance.MaxResidentChunks = ServerConfiguration.GetOrUpdateSetting(
|
|
"pathfinding.maxResidentChunks",
|
|
8192
|
|
);
|
|
|
|
PathfindRecorder.Configure();
|
|
|
|
CommandSystem.Register("PathCacheStats", AccessLevel.Administrator, OnPathCacheStats);
|
|
CommandSystem.Register("PathCacheClear", AccessLevel.Administrator, OnPathCacheClear);
|
|
CommandSystem.Register("PathBake", AccessLevel.Administrator, OnPathBake);
|
|
CommandSystem.Register("PathCacheSave", AccessLevel.Administrator, OnPathCacheSave);
|
|
CommandSystem.Register("PathCacheLoad", AccessLevel.Administrator, OnPathCacheLoad);
|
|
CommandSystem.Register("PathRecord", AccessLevel.Administrator, OnPathRecord);
|
|
AutoLoadAtStartup();
|
|
}
|
|
|
|
/// <summary>
|
|
/// Open Data/Pathfinding/<mapId>.swb as a lazy backing store for every map.
|
|
/// Reads only the header + chunk-offset index up front (~16 bytes per chunk);
|
|
/// individual chunk records are fetched on demand when the cache asks for them.
|
|
/// RAM stays bounded by MaxResidentChunks regardless of file size.
|
|
/// </summary>
|
|
private static void AutoLoadAtStartup()
|
|
{
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID);
|
|
}
|
|
}
|
|
|
|
[Usage("PathCacheStats")]
|
|
[Description("Reports StepCache resident-chunk count and hit/miss/eviction telemetry.")]
|
|
private static void OnPathCacheStats(CommandEventArgs e)
|
|
{
|
|
var stats = StepCache.Instance.GetStats();
|
|
var from = e.Mobile;
|
|
|
|
from.SendMessage($"StepCache: {stats.ResidentChunks} chunks resident");
|
|
from.SendMessage($" builds={stats.BuildsTotal} hits={stats.Hits}");
|
|
from.SendMessage($" miss(notBuilt)={stats.MissesNotBuilt} miss(dirty)={stats.MissesDirtyRebuild}");
|
|
from.SendMessage($" fallthru(multiZ)={stats.FallthroughMultiZ} fallthru(offMap)={stats.FallthroughOffMap} fallthru(srcZ)={stats.FallthroughSourceZMismatch}");
|
|
from.SendMessage($" evictions(lruCap)={stats.EvictionsByLruCap}");
|
|
}
|
|
|
|
[Usage("PathCacheClear")]
|
|
[Description("Drops all StepCache resident chunks and zeros the telemetry counters.")]
|
|
private static void OnPathCacheClear(CommandEventArgs e)
|
|
{
|
|
var residentBefore = StepCache.Instance.GetStats().ResidentChunks;
|
|
StepCache.Instance.Clear();
|
|
e.Mobile.SendMessage($"StepCache cleared: {residentBefore} chunks dropped, counters reset.");
|
|
}
|
|
|
|
[Usage("PathBake [mapId]")]
|
|
[Description("Walks every chunk of the given map (or all loaded maps) building the full static step cache, then saves it to Data/Pathfinding/<mapId>.swb so a future boot has zero first-pathfind latency. WARNING: blocks the game loop for several seconds and transiently uses hundreds of MB per map — run during maintenance, not peak hours.")]
|
|
private static void OnPathBake(CommandEventArgs e)
|
|
{
|
|
var from = e.Mobile;
|
|
int? only = e.Arguments.Length > 0 && int.TryParse(e.Arguments[0], out var parsed) ? parsed : null;
|
|
|
|
from.SendMessage("PathBake: building the static step cache. The server will pause briefly per map...");
|
|
|
|
var totalChunks = 0;
|
|
var totalMaps = 0;
|
|
var sw = System.Diagnostics.Stopwatch.StartNew();
|
|
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal || (only.HasValue && map.MapID != only.Value))
|
|
{
|
|
continue;
|
|
}
|
|
|
|
// BakeMap walks the whole map (building every chunk) and writes the .swb. The
|
|
// chunks are left resident afterward; drop them so peak memory is bounded to one
|
|
// map at a time and the post-command footprint returns to the LRU cap.
|
|
var written = StepCache.Instance.BakeMap(map.MapID, PathFor(map.MapID));
|
|
StepCache.Instance.ClearResidentChunks();
|
|
|
|
if (written > 0)
|
|
{
|
|
totalChunks += written;
|
|
totalMaps++;
|
|
from.SendMessage($" map {map.MapID}: {written} chunks → {PathFor(map.MapID)}");
|
|
}
|
|
}
|
|
|
|
sw.Stop();
|
|
|
|
if (totalMaps == 0)
|
|
{
|
|
from.SendMessage(only.HasValue ? $"PathBake: map {only.Value} not loaded." : "PathBake: no maps to bake.");
|
|
return;
|
|
}
|
|
|
|
// Reopen the freshly written files as lazy backing stores so they're usable now
|
|
// without a restart (resident memory stays bounded by the LRU cap).
|
|
AutoLoadAtStartup();
|
|
from.SendMessage($"PathBake: {totalChunks} chunks across {totalMaps} map(s) in {sw.Elapsed.TotalSeconds:F1}s; lazy readers reopened.");
|
|
}
|
|
|
|
[Usage("PathCacheSave")]
|
|
[Description("Persists resident StepCache chunks for every loaded map to Data/Pathfinding/<mapId>.swb.")]
|
|
private static void OnPathCacheSave(CommandEventArgs e)
|
|
{
|
|
var totalChunks = 0;
|
|
var totalMaps = 0;
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
var path = PathFor(map.MapID);
|
|
var written = StepCache.Instance.SaveToFile(path, map.MapID);
|
|
if (written > 0)
|
|
{
|
|
totalChunks += written;
|
|
totalMaps++;
|
|
e.Mobile.SendMessage($" map {map.MapID}: {written} chunks → {path}");
|
|
}
|
|
}
|
|
e.Mobile.SendMessage($"StepCache saved: {totalChunks} chunks across {totalMaps} map(s).");
|
|
}
|
|
|
|
[Usage("PathCacheLoad")]
|
|
[Description("Opens Data/Pathfinding/<mapId>.swb as a lazy backing store for every map. Chunks are fetched on demand, so RAM stays bounded by the LRU cap regardless of file size.")]
|
|
private static void OnPathCacheLoad(CommandEventArgs e)
|
|
{
|
|
var openedMaps = 0;
|
|
for (var i = 0; i < Map.Maps.Length; i++)
|
|
{
|
|
var map = Map.Maps[i];
|
|
if (map == null || map == Map.Internal)
|
|
{
|
|
continue;
|
|
}
|
|
if (StepCache.Instance.TryOpenLazyReader(PathFor(map.MapID), map.MapID))
|
|
{
|
|
openedMaps++;
|
|
}
|
|
}
|
|
e.Mobile.SendMessage(
|
|
$"StepCache: opened {openedMaps} map(s) for lazy loading (total readers open: {StepCache.Instance.OpenLazyReaderCount})."
|
|
);
|
|
}
|
|
|
|
[Usage("PathRecord [on|off|flush|status]")]
|
|
[Description("Toggles PathfindRecorder. With no arg, reports state. 'on' enables JSONL capture of every Find call; 'off' disables and flushes; 'flush' forces a buffer flush without disabling.")]
|
|
private static void OnPathRecord(CommandEventArgs e)
|
|
{
|
|
var arg = (e.Arguments.Length > 0 ? e.Arguments[0] : "status").ToLowerInvariant();
|
|
var from = e.Mobile;
|
|
switch (arg)
|
|
{
|
|
case "on":
|
|
{
|
|
PathfindRecorder.SetEnabled(true);
|
|
from.SendMessage(PathfindRecorder.Enabled
|
|
? $"PathRecord: ON, writing to {PathfindRecorder.OutputPath}"
|
|
: "PathRecord: enable failed (see server log)");
|
|
break;
|
|
}
|
|
case "off":
|
|
{
|
|
PathfindRecorder.SetEnabled(false);
|
|
from.SendMessage("PathRecord: OFF");
|
|
break;
|
|
}
|
|
case "flush":
|
|
{
|
|
PathfindRecorder.Flush();
|
|
from.SendMessage($"PathRecord: flushed ({PathfindRecorder.RecordsWritten} records this session)");
|
|
break;
|
|
}
|
|
default:
|
|
{
|
|
from.SendMessage(
|
|
$"PathRecord: {(PathfindRecorder.Enabled ? "ON" : "OFF")}, "
|
|
+ $"path={PathfindRecorder.OutputPath}, records={PathfindRecorder.RecordsWritten}"
|
|
);
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
}
|