Commit graph

11 commits

Author SHA1 Message Date
Kamron Batman
a8ca82738d
feat(pathfinding): JSONL recorder + public bake helpers (#2449)
## Summary

Adds two pieces of pathfinding tooling on top of PR #2448's lazy `.swb` infrastructure:

- **`PathfindRecorder`** — admin-toggled JSONL telemetry capture; one record per `BitmapAStarAlgorithm.Find` call. Output format matches the BDN harness corpus, so production traffic can be captured and replayed in benchmarks without an adapter.
- **Public bake helpers on `StepCache`** — `ComputeLiveTileDataHash`, `TryReadTileDataHashFromFile`, `BakeMap`, `ClearResidentChunks`. Lets the benchmark project (and any future bake utility) drive cache fill + persist without exposing internal types.

The companion BDN harness update lives in [ModernUO-Benchmarks#kb/pathfinding-pr4-bench](https://github.com/modernuo/ModernUO-Benchmarks/tree/kb/pathfinding-pr4-bench): porting `Benchmarks/PathfindInGame/` from the `kb/ai_pathfinding` branch to the API shipped in #2446–#2448.

## What's in this PR

### `PathfindRecorder` (`PathfindRecorder.cs`)
- Holds a single `StreamWriter` open while recording; its internal buffer absorbs per-record writes without per-call `File.AppendAllText`.
- Single `bool` check on the hot path; cheap when disabled.
- Disabling flushes + disposes; an IO failure during write also disables the recorder.
- Server config:
  - `pathfinding.recorder.enable` — bool, default `false`. Read on boot via `GetOrUpdateSetting`.
  - `pathfinding.recorder.path` — default `<basedir>/Data/Pathfinding/recordings/pathfinds.jsonl`.
- Hooked into `BitmapAStarAlgorithm.Find` — runs once per call, does nothing when disabled.
- Admin command: `[PathRecord [on|off|flush|status]` (default `status`).

### Public cache helpers
- `static ulong StepCache.ComputeLiveTileDataHash()` — wraps the file module's hash function for staleness checks.
- `static bool StepCache.TryReadTileDataHashFromFile(string, out ulong)` — peeks at a `.swb` file's hash field (20 bytes).
- `int StepCache.BakeMap(int, string)` — walks every chunk in the map, populates resident set, saves. Offline / fixture use; blocks for many seconds on a full-map walk.
- `void StepCache.ClearResidentChunks()` — drops chunks + zeros counters but keeps lazy readers open. Lets benchmark loops measure "first query after boot" cost across iterations without the lazy-reader reopen overhead.
2026-05-06 13:06:28 -07:00
Kamron Batman
7c9215d97c
feat(pathfinding): lazy .swb backing store for the step cache (#2448)
## Summary

Adds a binary disk format + lazy reader so the step cache can warm-start from a precomputed file without paying chunk-build cost on the first pathfind through a region. **Resident memory stays bounded by `MaxResidentChunks` regardless of file size** — opening a `.swb` reads only the header + chunk-offset index (~16 bytes per indexed chunk), and individual chunks are seeked + deserialized only when `ResolveMissingChunk` asks for them.

The lazy design (vs. an eager bulk load): a 250 MB bake on a RAM-constrained shard never materializes more than the LRU cap (~40 MB at the default 8192-chunk cap), and unwanted regions never enter memory at all.

Builds on PR #2447.

## What changed

- **`StepCacheFile`** — binary reader/writer module. Writer emits header → chunks (offsets recorded) → index trailer, then patches the header's `IndexOffset` field. Reader is `OpenForLazy(path)` returning a `LazyReader` that holds an open `FileStream` + offset dictionary.
- **`StepCacheFile.LazyReader`** — `TryReadChunk(chunkX, chunkY)` does a single seek + bulk read for one record. `Dispose` releases the underlying stream. Files are opened with `FileShare.Read | FileShare.Delete` so admin tooling can replace them.
- **TileData fingerprint via XxHash3.** The `.swb` header carries a hash of `LandTable + ItemTable` flags. Load rejects any file whose hash doesn't match the running server. Computed via `HashUtility.ComputeHash64` (engine-blessed hasher) — adds a `ReadOnlySpan<byte>` overload alongside the existing `ReadOnlySpan<char>` one for parity.
- **`StepCache.SaveToFile(path, mapId)`** — writes resident chunks for the given map.
- **`StepCache.TryOpenLazyReader(path, mapId)`** — opens the file, validates header, holds the reader for the map's lifetime.
- **`StepCache.ResolveMissingChunk`** — now consults the lazy reader before invoking the runtime baker. A loaded chunk whose `BuiltMultisVersion` doesn't match the live sector falls through to the baker (snapshot was made before a multi was added/removed in that sector).
- **`StepCache.Clear` closes lazy readers.** Test cleanup can delete `.swb` files cleanly.
- **Auto-load at startup.** `PathCacheCommands.Configure()` opens `Data/Pathfinding/<mapId>.swb` as a lazy reader for every map.
- **`[PathCacheSave`** / **`[PathCacheLoad`** — admin commands for the same workflow.
- **`pathfinding.maxResidentChunks` shard-tunable.** Read from `server.cfg` at boot via `ServerConfiguration.GetOrUpdateSetting` (default 8192 ≈ 40 MB). Small shards can tune down; large shards with substantial bakes can tune up to reduce eviction churn. Default is written back to `server.cfg` on first boot, matching the engine pattern used by other settings.

## File layout (v1)

```
Header (48 bytes):
  u32  Magic           = 0x42575300 ('SWB\0')
  u32  Version         = 1
  u32  MapId
  u64  TileDataHash    XxHash3 over LandTable + ItemTable flags (HashUtility)
  u64  BakeTimestamp   informational
  u32  ChunkCount
  u64  IndexOffset     file position where the chunk index begins

Chunk records (fixed size, ~5,393 bytes each, +32 if multi-Z):
  u16  ChunkX
  u16  ChunkY
  u32  BuiltMultisVersion
  u8   HasMultiZ
  byte WalkMask[256], WetMask[256]
  sbyte SourceZ[256], WalkZN..WalkZNW[256], SwimZN..SwimZNW[256]
  [byte MultiZCells[32] when HasMultiZ == 1]

Index trailer (16 × ChunkCount bytes):
  (u64 chunkKey, u64 fileOffset)
```

## Memory math

| Scenario | Disk file | RAM at boot | Notes |
|---|---|---|---|
| Empty / no `.swb` files | — | 0 | Silent; cache builds on demand. |
| Admin-curated towns (5K chunks) | 25 MB | 0 + per-query | Index ≈ 80 KB. Resident grows to the configured cap under steady-state queries. |
| Full-map bake (50K chunks) | 250 MB | 0 + per-query | Index ≈ 800 KB. Same configured cap. Cold areas never load. |
| All 5 maps fully baked | 1.25 GB | 0 + per-query | Index ≈ 4 MB total. Same configured cap. |

## Hash choice (FNV-1a → XxHash3)

The original draft used inlined FNV-1a-64. Switched to XxHash3 via `HashUtility`:

- ~30× faster on this workload (~30 GB/s SIMD vs ~2 GB/s byte-by-byte). Boot-time only, so absolute saving is microseconds — the real wins are elsewhere.
- Stronger collision resistance and distribution.
- Drops ~25 lines of inlined hash code; matches the rest of the codebase's hashing pattern.
- Hash is stable as long as `HashUtility`'s `xxHash3Seed` constant doesn't change (already marked `// DO NOT CHANGE THIS NUMBER`).
2026-05-06 01:32:44 -07:00
Kamron Batman
e1e1a7c640
fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Kamron Batman
ebaf104935
chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Kamron Batman
c75514cc04
feat: Updates to .NET 9 (#1984)
### Summary

* Bumps to .NET 9 with updated dependencies
* Comparing a value type against null is no longer allowed
* CI/CD now uses the version specified in global.json
* Serialization generator updated to .NET 9 with bug fixes, fixes to turkish language, and parallelization
2024-12-08 10:16:34 -08:00
Kamron Batman
1c10b6dbed
fix: Adds support for ROS to HashUtility (#1740) 2024-04-25 00:42:10 -07:00
Kamron Batman
d5253c2bae
chore: Cleans up unused imports (#1608) 2023-11-21 12:32:51 -08:00
Kamron Batman
1769d47ba5
fix: Removes Standart XxHash in favor of the built in one (#1598) 2023-11-17 16:11:35 -08:00
Kamron Batman
8389bfacfe
chore: Updates copyright (#1448) 2023-08-09 09:09:26 -07:00
Kamron Batman
50d52fb25e
fix: Makes gump type id stable (#1358) 2023-03-02 19:29:57 -08:00
Kamron Batman
e1e30998ba
fix: Adds ReadType/Write(Type) and improves type referencing (#1172)
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`

View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1

## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.

### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8

### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">

## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
|               Method |     Mean |    Error |   StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
|      BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us |    8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us |         - |
```

TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
2022-10-11 22:17:22 -07:00