## Summary Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them. The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all. Builds on PR #2447. ## What changed - **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary. - **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them. - **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity. - **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map. - **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime. - **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector). - **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly. - **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map. - **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow. - **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings. ## File layout (v1) ``` Header (48 bytes): u32 Magic = 0x42575300 ('SWB\0') u32 Version = 1 u32 MapId u64 TileDataHash XxHash3 over LandTable + ItemTable flags (HashUtility) u64 BakeTimestamp informational u32 ChunkCount u64 IndexOffset file position where the chunk index begins Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z): u16 ChunkX u16 ChunkY u32 BuiltMultisVersion u8 HasMultiZ byte WalkMask[256], WetMask[256] sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256] [byte MultiZCells[32] when HasMultiZ == 1] Index trailer (16 × ChunkCount bytes): (u64 chunkKey, u64 fileOffset) ``` ## Memory math | Scenario | Disk file | RAM at boot | Notes | |---|---|---|---| | Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. | | Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. | | Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. | | All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. | ## Hash choice (FNV-1a → XxHash3) The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`: - ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere. - Stronger collision resistance and distribution. - Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern. - Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
118 lines
3.8 KiB
C#
118 lines
3.8 KiB
C#
/*************************************************************************
|
|
* ModernUO *
|
|
* Copyright 2019-2026 - ModernUO Development Team *
|
|
* Email: hi@modernuo.com *
|
|
* File: HashUtility.cs *
|
|
* *
|
|
* This program is free software: you can redistribute it and/or modify *
|
|
* it under the terms of the GNU General Public License as published by *
|
|
* the Free Software Foundation, either version 3 of the License, or *
|
|
* (at your option) any later version. *
|
|
* *
|
|
* You should have received a copy of the GNU General Public License *
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
|
|
*************************************************************************/
|
|
|
|
using System;
|
|
using System.IO.Hashing;
|
|
using System.Numerics;
|
|
using System.Runtime.InteropServices;
|
|
|
|
namespace Server;
|
|
|
|
public static class HashUtility
|
|
{
|
|
// *************** DO NOT CHANGE THIS NUMBER ****************
|
|
// * Computed hashes might be serialized against this seed! *
|
|
// **********************************************************
|
|
private const ulong xxHash3Seed = 9609125370673258709ul; // Randomly generated 64-bit prime number
|
|
private const uint xxHash1Seed = 665738807u; // Randomly generated 32-bit prime number
|
|
|
|
[ThreadStatic]
|
|
private static XxHash3 _xxHash3;
|
|
|
|
[ThreadStatic]
|
|
private static XxHash32 _xxHash32;
|
|
|
|
public static ulong ComputeHash64(ReadOnlySpan<char> str)
|
|
{
|
|
if (str.Length == 0)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
var hasher = _xxHash3 ??= new XxHash3(unchecked((long)xxHash3Seed));
|
|
hasher.Append(MemoryMarshal.Cast<char, byte>(str));
|
|
|
|
var result = hasher.GetCurrentHashAsUInt64();
|
|
hasher.Reset();
|
|
|
|
return result;
|
|
}
|
|
|
|
public static ulong ComputeHash64(ReadOnlySpan<byte> bytes)
|
|
{
|
|
if (bytes.Length == 0)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
var hasher = _xxHash3 ??= new XxHash3(unchecked((long)xxHash3Seed));
|
|
hasher.Append(bytes);
|
|
|
|
var result = hasher.GetCurrentHashAsUInt64();
|
|
hasher.Reset();
|
|
|
|
return result;
|
|
}
|
|
|
|
public static uint ComputeHash32(ReadOnlySpan<char> str)
|
|
{
|
|
if (str == ReadOnlySpan<char>.Empty)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
var hasher = _xxHash32 ??= new XxHash32(unchecked((int)xxHash1Seed));
|
|
hasher.Append(MemoryMarshal.Cast<char, byte>(str));
|
|
|
|
var result = hasher.GetCurrentHashAsUInt32();
|
|
hasher.Reset();
|
|
|
|
return result;
|
|
}
|
|
|
|
public static unsafe int GetNetFrameworkHashCode(this string? str)
|
|
{
|
|
if (str == null)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
fixed (char* src = &str.GetPinnableReference())
|
|
{
|
|
uint hash1 = (5381 << 16) + 5381;
|
|
var hash2 = hash1;
|
|
|
|
var ptr = (uint*)src;
|
|
var length = str.Length;
|
|
|
|
while (length > 2)
|
|
{
|
|
length -= 4;
|
|
// Where length is 4n-1 (e.g. 3,7,11,15,19) this additionally consumes the null terminator
|
|
hash1 = (BitOperations.RotateLeft(hash1, 5) + hash1) ^ ptr[0];
|
|
hash2 = (BitOperations.RotateLeft(hash2, 5) + hash2) ^ ptr[1];
|
|
ptr += 2;
|
|
}
|
|
|
|
if (length > 0)
|
|
{
|
|
// Where length is 4n-3 (e.g. 1,5,9,13,17) this additionally consumes the null terminator
|
|
hash2 = (BitOperations.RotateLeft(hash2, 5) + hash2) ^ ptr[0];
|
|
}
|
|
|
|
return (int)(hash1 + hash2 * 1566083941);
|
|
}
|
|
}
|
|
}
|