## Summary
`Mobile.DamageEntries` was a `List<DamageEntry>` allocated for every mobile, including the ~99% that never take damage. It is now an inline `ValueLinkList<DamageEntry>` (24 bytes in the `Mobile` object, no separate allocation) ordered least recent → most recent.
- `DamageEntry` implements `IValueLinkListNode<DamageEntry>`.
- `RegisterDamage` moves the entry to the tail in O(1) instead of `Remove` + `Add` on a list.
- Expired entries are always a head prefix, so pruning walks from the head and stops at the first live entry. The `DamageEntries` getter prunes on access.
- `DamageEntries` is exposed as `ref readonly`; enumerate with `foreach` or `.ByDescending()`. Mutation goes through `RegisterDamage` / `ClearDamageEntries`.
- `BaseCreature.GetLootingRights` and `BaseCreature.ComputeBonusDamage` take `in ValueLinkList<DamageEntry>`; all callers compile unchanged. Files that `foreach` over `DamageEntries` need `using Server.Collections;` for the enumerator extension.
- RunUO migration docs (`dev-docs/runuo-migration-docs/09`, `11`) and the `migrate-items-mobiles` skill document the change.
Saves one object and 16 bytes per mobile (~8 MB and 500k gen2 objects on a 500k world). Second of three PRs from the lazy per-mobile collections design (first: #2604). Branched from `main`; the two diffs touch disjoint hunks of `Mobile.cs`.
## Breaking change
- `Mobile.DamageEntries` is no longer a `List<DamageEntry>`. Indexing, `.Clear()`, `.Add()`, `.Remove()` no longer compile; use `foreach`, `.ByDescending()`, `.Count`, `ClearDamageEntries()`, and `RegisterDamage`. Calling a `ValueLinkList` mutator on the `ref readonly` property compiles but operates on a copy while still unlinking the real nodes; do not.
- `BaseCreature.GetLootingRights` and `BaseCreature.ComputeBonusDamage` signatures changed to `(in ValueLinkList<DamageEntry>, …)`.
Save format is untouched: damage entries are not serialized.
## Behavior
Recency order, `allowSelf`, tie-breaking in `FindMostTotal`/`FindLeastTotal` (most recent wins), `Responsible` accounting, and loot-rights ordering are unchanged and covered by the new `DamageEntryTests` and `LootingRightsTests`.
## Testing
- `dotnet build -c Release` clean.
- New `DamageEntryTests` and `LootingRightsTests` plus full `Server.Tests` and `UOContent.Tests`.
Fixes walk-up aggro latency (up to a full 10 s of obliviousness) and hardens the reacquire gate so no state can silence acquisition, while turning `AcquireOnApproach` into the reaction-time knob for future per-creature intelligence tuning.
### Why
`AcquireFocusMob` re-armed the 10 s `ReacquireDelay` **before** scanning, success or failure. A creature that scanned an empty room was blind for 10 s to a player walking up — walk-up aggro latency was uniform in 0..10 s. Waking from sector sleep stacked the AI timer's 0–3 s construction stagger on top. And `NextReacquireTime` is not serialized: on hosts whose tick counter starts negative (GCP pass-through), the 0 default blocked **all** acquisition shard-wide after a restart until the counter crossed zero.
### What
**Event-driven reaction — `AcquireOnApproachDelay` (the intelligence gradient)**
- The paragon `AcquireOnApproach` bool becomes a `TimeSpan` on every creature: an enemy moving inside `AcquireOnApproachRange` (10 for all creatures — on-screen reactive aggro; the periodic scan keeps the wide `RangePerception` sweep) *clamps* the next scan to at most the delay. Repeated steps cannot shorten it further — one scan per delay period, not per step or think.
- `Zero` (paragons) also prods the AI timer: the ranked scan engages within a wheel turn — the old snap, minus the special-cased engage path. The target now comes from the normal FightMode ranking instead of whichever mobile happened to move, and the `Combatant == null` guard stops re-engage spam.
- The 2 s default reads as "took a beat to notice you"; larger values are dumber; `ReacquireDelay` alone is the oblivious floor. Mover checks are the approach logic's `IsEnemy` + `CanBeHarmful` (so pets count and hidden movers are excluded via `CanSee`), with `IsEnemy` first to cheaply reject same-team wild creatures wandering past. The check rides the `OnMovement` callback every step already pays for — no polling added.
**Gate correctness**
- Every scan re-arms the full `ReacquireDelay`, success or failure (classic semantics; reaction time is the approach path, not the poll).
- Self-healing by construction: a deadline further out than `ReacquireDelay` is an illegal state and reads as open — no wedged or wrapped value can silence acquisition beyond one delay period.
- `NextReacquireTime` is seeded from a live tick on deserialize (the GCP negative-tick blackout).
**AI timer wake**
- Activation (sector wake, spawn, resurrection) starts within a 0–256 ms spread instead of the 0–3 s construction stagger, which read as lag.
- The stagger's real job — keeping same-speed cohorts out of lock-step (the RunUO town artifact) — is now a zero-mean ±period/8 jitter on each **idle** think, so phases random-walk apart within seconds and can never re-lock. Instrumentation showed why a one-shot spread can't do this job: the timer wheel fires within ±1 ms, so with 10 creatures on a 500 ms period some pair collides on nearly the same phase ~75% of the time (birthday paradox) and then steps in the same loop iteration *forever*. Jitter is scoped to passive speed: engaged cadence stays exact, since pursuit timing anchors to real step times.
**Debug**
- The `AcquireFocusMob` scan message no longer re-arms the shared 5 s debug cooldown, which swallowed every AI's "I have detected X" transition line.
**API change** for custom scripts: `AcquireOnApproach` (bool) → `AcquireOnApproachDelay` (TimeSpan). Documented in `content-patterns.md` § Target Acquisition, `runuo-migration-docs/09` + `11`, and the migration skill checklist.
### Tests
`AcquisitionTests`: both scan outcomes honor `ReacquireDelay`; a 60 s-wedged gate still acquires; enemy movement clamps the deadline (same-team wild movers and out-of-range movers ignored); repeated movement cannot shorten below the delay; `Zero` opens the gate and prods without a direct engage. Full suite: 755 UOContent green.
Stacked on #2594. Fixes jerky creature movement (lich / Fast-bucket melee chases) by choosing the client animation flag from the actual step pace instead of a caller-supplied `run` argument, and fixes three step-pacing defects in the move budget found while verifying it with paired server/client traces.
### Why
The `Direction.Running` bit does nothing for creatures server-side (`Mobile.OnMove` reads it only for the player throttle and stealth reveal). Its whole effect is on the client, which animates each step over a fixed time selected by that bit: walk 400 ms / run 200 ms on foot, 200 / 100 ms mounted. ClassicUO queues up to 5 steps and *drops* the sixth, so a creature stepping every 300 ms while flagged as walking backs the queue up until it snaps forward — the observed jerk.
The `run` argument never carried the one fact that matters (the step interval). RunUO passed `true` in combat / `false` for pets and gated it on `dist > 5`; #2271 flipped every combat site to `false`; pets passed `currentDistance > 2`. None of that is a coherent signal.
### What
**Pace-derived run flag**
- `BaseAI.ShouldRun()`: run iff the effective step delay (move clock + badly-hurt inflation) is shorter than `Movement.WalkFootDelay` / `WalkMountDelay` (mounted or flying) — with a continuity rule: an *isolated* step (taken after standing at least a walk interval) goes out as a walk, because the client renders each step alone and a lone run-flagged step is a 200 ms dart. Only a continuing cadence flags run; a true sprinter (pace under the run interpolation) always runs, since a walk-rendered first step would flood the client's 5-step queue. This reproduces RunUO's close-in feel (its `dist > 5` gate) from first principles.
- `DoMoveImpl` stamps the bit; it is the single place the flag is set.
- `run` removed from `MoveTo`, `WalkMobileRange`, `ApproachTarget`, `MoveToPoint`, `MoveToWithGroup`, `MoveToWithCollisionAvoidance`, the move intent, and `PathFollower.Follow`. All 35 call sites updated. **API change** for custom scripts — documented in the RunUO migration docs (`09-items-mobiles-creatures.md`, `11-api-reference.md`) and `content-patterns.md` § Creature Speeds.
**Move-budget pacing fixes** (each confirmed by UTC-aligned server/client step traces)
- A stall no longer banks catch-up steps: the budget's snap-to-now released up to three steps in ~300 ms when a creature resumed chasing after standing beside its target — rendered as a teleport.
- Debt accrual removed entirely: a step landing sub-period late (think-grid vs budget misalignment during reactive mirroring) kept the remainder and fired a follow-up ~100 ms later — a dart pair. `ConsumeMoveBudget` now paces every step from when it was actually taken; in continuous pursuit the move-wake lands within wheel resolution of the deadline, so the cost is single-digit-ms drift.
- Net effect: a creature can never step faster than its pace, verified across a full chase session (zero sub-pace steps; metronomic 350 ms cadence for a 0.3 s lich).
- Test fixture now runs `Movement.Configure()` (the walk delays were 0 in tests).
### Accepted trade-off
Animal (LOW group) bodies without a run animation slide on their stand frames when flagged as running. Most are slow enough to stay flagged as walking; the client-side fallback is in ClassicUO/ClassicUO#1930.
### Tests
`RunFlagTests`: foot thresholds (0.3 / 0.125 run; 0.4 / 0.45 / 1.05 walk), flying uses the mount threshold, badly-hurt inflation flips a 0.35 s creature back to walk, a real `DoMove` stamps the bit, isolated steps drop to walk (sprinters keep running), a stall restarts the cadence with no banked steps, and a late step earns no quicker follow-up. Full suite: 837 Server + 747 UOContent green.
Closes#2593. Closes#2595.
Two related pet-AI fixes: the post-#2591 pacing/wake regression (#2593), and the guard order silently converting to Attack during combat (#2595). Root-cause analyses are in the issues.
## #2593 — pets follow slowly; stale AITimer wakes
**Why pets slowed:**
- The per-step budget grew from **half a think interval** (`CurrentSpeed * 500`) to the full RunUO-parity move table (`CurrentMoveSpeed * 1000`). Medium-bucket pets (Horse, Dog, most tamables): passiveMove **1.05s/step**.
- Pet order speed depended on stale `Warmode`: `HandleGuardOrder` set it once, but `OnCombatantChange` clears it whenever the combatant drops, so obedience ran active or passive **by combat history** — usually passive. Net: Guard/Come at ~1.05s/step (~2.1x slower than pre-#2591), vs a player running at 0.1–0.2s/step.
- The AITimer never rescheduled its pending wheel entry: the wheel reads `Interval` only after the next fire, so a speed-up or a fresh order (`Activate()` no-ops while running) waited out the stale wake — up to a full passive think, stacked on the residual move budget on Guard → Follow.
**What changed:**
- **Order handlers own obedience speed** (RunUO `OnCurrentOrderChanged`/`DoOrder*` parity, re-derived continuously): issuing a movement order (Come/Follow/Guard/Attack) sets the **active** think clock, resting orders (Stay/None/Transfer) set passive, and the guard/follow peaceful branches write **RunUO's AOS `CurrentSpeed = 0.1` sprint** — RunUO's guard else-branch had the identical write as follow. The bespoke 0.1 fuses to both clocks through #2591's existing classification, so `CurrentMoveSpeed` stays **pure herding + classification** with no obedience special case, and `DoMoveImpl`'s per-step flip skips obeying pets (their handler owns the pace) and loses its old follow-only 0.1 write. Combat still re-derives organically via warmode/combatant.
- **`AITimer`**: tracks the pending wake and reschedules (`Stop`, `Delay` = remaining, `Start`) when a speed-up or fresh order moves the earliest deadline up; changes inside a tick still flow through `ScheduleNext`. New `Prod()` wakes the AI immediately on player commands — including from a stopped timer, so stable claims no longer wait out the random construction stagger. Sector/spawn wakes keep the stagger. Spam-safe: a prodded think grants reaction, never action — steps/swings/casts/abilities are gated by their own budgets and timers.
The residual move budget is deliberately **not** cleared on order change — that would let order-spam macros grant free steps. Deadline changes reschedule the timer; rate changes take effect at the next deadline computation.
## #2595 — Guard order converts to Attack during combat
**Why:** `FindCombatant()` set `ControlOrder = OrderType.Attack` when engaging, so a guarding pet left the Guard order for the whole fight: OPL tags wiped (pet `1080078` + master `501129`), no retargeting (`DoOrderAttack` locks its target), `TeleportPets` left the pet behind on recall/gate, and every engage→kill→resume cycle replayed the guard flourish.
**What changed:**
- **`FindGuardTarget()`** (was `FindCombatant`): a pure selector — prefers the aggressor **closest to the master** (RunUO guard parity, dynamic retargeting to protect the owner), keeps the current combatant unless a strictly closer one exists, and never mutates order state. `DoOrderGuard` engages through it while **staying in Guard** the whole fight.
- **Persistent-order semantics** (the ModernUO improvement over RunUO): an explicit `all attack` completes → `ResumePersistentOrder()` returns to Guard → the guard scan engages remaining threats in-order. The Attack-chaining fallback (`FightMode.Closest/Aggressor`) now applies only to non-guard persistent orders. Resuming Guard no longer replays the sound/"is now guarding you" message.
- **Peaceful guard stands down deterministically** (`Warmode`/`Combatant`/`FocusMob` cleared) and returns to the master at the RunUO sprint (see above); at the master's side it stays organically active.
- **`WalkMobileRange` honors the caller's run flag** (the internal hardcoded `dist > 5` gate silently overrode it). Run is animation-only server-side; the only callers passing anything but `false` — follow, guard, clone — gate on their own thresholds.
## Resulting behavior (Medium-bucket pet)
| Scenario | Broken | This PR |
|---|---|---|
| Guard trailing master (AOS) | ~1.05s/step, think-grid quantized | 0.1s/step sprint (RunUO parity), smooth move wakes |
| Guard during combat | order flips to Attack; tags lost; no retarget; left behind on recall | stays Guard; retargets to master's closest aggressor; teleports with master |
| `all attack` while guarding | resume spams guard flourish per kill; chains into Attack | resumes Guard silently; guard scan takes over |
| Come / friend-follow | 1.05s/step | activeMove 0.45s/step (≈ pre-#2591 feel) |
| Guard → Follow reaction | up to ~1.5s dead time | think within one wheel turn |
| Follow master (AOS sprint) | 0.1s/step | 0.1s/step (unchanged) |
| Wild creature chase | RunUO-parity move table | unchanged |
Also documents two contracts this work leaned on: the `ControlOrder` setter deliberately fires on every assignment (a reissued order is a command — retarget/break-off/re-anchor), and `OnThink`/`MonsterAbility` must be excess-call tolerant (`dev-docs/content-patterns.md` § OnThink: the excess-call contract).
## Testing
- Full suite passes (1570: 837 Server + 733 UOContent).
- `PetPacingTests`: order-issue think-clock parity, follow-master sprint via Obey, guard organically active at the master's side, combat-chase and herding boundaries, plus two deterministic timer-wheel tests (8ms-lockstep slicing) proving a fresh order and a mid-wait speed-up wake the AI promptly.
- `GuardOrderTests`: engage keeps the Guard order; retargets to the aggressor closest to the master; explicit attack resumes Guard without chaining into Attack; peaceful guard stands down. Setup self-validates LOS/terrain.
- `GuardFollowTests`: guard-following registers a move intent, steps toward the master, sprints at 0.1 under AOS (per-step flip must not undo it), and runs active pre-AOS.
- All behavioral tests were written first and failed for the documented reasons.
Splits creature speed into two clocks so movement pace can be tuned without touching reaction time:
- **Think clock** — `ActiveSpeed`/`PassiveSpeed`/`CurrentSpeed`: seconds per AI decision. Unchanged in meaning, storage, and cadence.
- **Move clock** — `ActiveMoveSpeed`/`PassiveMoveSpeed` (+ resolved `CurrentMoveSpeed`): seconds per step. `0` = inherit the matching think value.
### How
- Move speeds come from optional `activeMove`/`passiveMove` in `npc-speeds.json`, are `[props`-tunable per instance (set `0` to re-inherit), and serialize (BaseCreature v22).
- `SetSpeed()` keeps its legacy one-clock semantics — sets the think clock **and clears move overrides** — so existing callers cannot half-configure a creature. `SetMoveSpeed()`/`ClearMoveSpeed()` configure movement explicitly; `ScaleMoveSpeed()` scales overrides for buffs.
- `CurrentMoveSpeed` is derived by classifying `CurrentSpeed`: a verbatim active/passive think value maps to the matching move value; a bespoke pace written directly (mount boosts, follow sprint) stays fused to both clocks. External `CurrentSpeed` writers need no changes.
- `AITimer` schedules the earlier of the two deadlines. Decisions run at the think cadence exactly as before; while a pursuit/investigation is live, the timer also wakes when the movement budget elapses and advances one step with no decisions. Steps no longer snap to the think grid, so any step delay paces smoothly on the 8ms wheel. A blocked creature schedules no move wakes.
- The movement budget is RunUO's `m_NextMove` accumulate-and-clamp at a full step, so long-run pacing averages `CurrentMoveSpeed` exactly.
### Behavior changes
- **`npc-speeds.json` buckets get RunUO `TransformMoveDelay`-parity move values**: creatures step at RunUO pace while thinking/reacting at current speed. The situational +0.1/+0.2 offsets are deliberately omitted.
- **Existing saves migrate on load**: a pre-v22 creature whose think speeds still match its npc-speeds entry (never hand-tuned) adopts the table's move values — worlds and pets pick up the new pacing without a respawn. Tuned creatures keep movement inheriting their think clock.
- **Paragons scale movement by `SpeedBuff` (1.2x)**: RunUO had no deliberate policy here — dividing by 1.2 knocked most speeds off `TransformMoveDelay`'s exact-equality table (raw pass-through, 2x+ faster), while 0.3/0.6 creatures landed back on it for ~1.33x. This applies the uniform 1.2x the buff always claimed. UnConvert snaps speeds back to exact table values within 1e-4 — /1.2 then ×1.2 drifts 0.45 and 0.9 by an ulp, which would read as hand-tuned (and defeat a future skip-table-conformant-values serialization pass); tuned speeds keep.
- **Herding paces the movement clock**: the old `CurrentSpeed` getter hack is gone. A herded creature walks at a fixed 0.3s/step — RunUO's forced pace, without its `TransformMoveDelay` inflation to 0.6 — so herding is never penalized by a slow creature. Thinking is untouched, and `CheckHerding` walks through `MoveToPoint`, so herded creatures path around obstacles.
- **Badly-hurt slowdown now inflates the step delay only** (RunUO parity), computed from the base each step. Previously it wrote `CurrentSpeed = CurrentSpeed + 0.05..0.15` back on every successful step — compounding unboundedly while hurt and slowing decisions too.
- Removes the vestigial `MoveSpeedMod` (never read, written, or serialized).
- With no bucket or per-instance move values, both clocks carry identical values and creatures pace as before.
### Testing
- Full suite passes (1557, including 12 new `MoveSpeedTests`: resolution classes, `SetSpeed` clearing, `0`-re-inherit, v22 round-trip with exact-consumption check, save migration adopt/skip, buff scale/snap, herding).
- In-game verified via local diagnostics build (per-step budget tracing): steady 700ms step cadence on a 0.3s think grid with one-step catch-up after idle, think grid unperturbed by move wakes.
## Summary
Brings every serialization-related doc, skill, and the CLAUDE.md rule in line with generator **v4** (adopted in #2586/#2587). No code changes.
**Updated surface, everywhere it was referenced:**
- `[SerializableFieldSaveFlag(order)]` / `[SerializableFieldDefault(order)]` → `[SaveFlag(nameof(Should), nameof(Default))]` on the field (second method optional).
- `[TimerDrift]` + `[DeserializeTimerField(order)]` → `[DeserializeTimer(nameof(Method), wallClock)]` on the field, with the anchored-time semantics spelled out: drifting by default (downtime preserves the remaining delay, idle saves byte-stable), `wallClock: true` for absolute deadlines, restart method invoked **only when a timer was running** (no sentinel), and the timer `MigrateFrom` pattern (`XxxNext`/`XxxDelay`) for wire-format changes.
- New `[SerializableField]` documentation: the real signature (the documented `saveIf` parameter never existed) plus the setter hooks — `allowFieldChange` (`bool Method(ref T value)`: coerce/veto before assignment) and `fieldChanged` (`void Method(T oldValue, T newValue)` after) — with the generated pipeline and the SG3015/SG3018 guardrails.
- `[SerializableProperty]` guidance narrowed to its remaining purpose: custom getters and setter semantics the hooks cannot express.
- `[AnchoredDateTime]` documented alongside `[DeltaDateTime]` (now marked legacy, with the version-bump warning for converting between them).
**Files:** `dev-docs/serialization.md`, `dev-docs/timers.md`, `dev-docs/claude-skills/modernuo-serialization.md`, `dev-docs/claude-skills/modernuo-timers.md`, `dev-docs/runuo-migration-docs/02-serialization.md`, `dev-docs/runuo-migration-docs/03-timers.md`, and a condensed v4 addition to CLAUDE.md rule 9.
**Example refresh:** the skill's `BagOfSending` "custom properties" example was itself converted in #2587 — it is now quoted in its real post-conversion form as the canonical hooks example; the real-examples list points at `BaseWeapon.cs` for custom getters and `BaseLight.cs` for the drifting-timer + `MigrateFrom` pattern.
Verified by grep: zero references to the removed v3 attribute names remain anywhere in `dev-docs/` or `CLAUDE.md`.
## Problem
`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.
## Result
Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:
| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |
**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.
## How
The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.
**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.
**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.
## Configuration
| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |
## Diagnostics (compiled out by default)
`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.
## Verification
- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.
## Notes
The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
## Why
An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.
## What the measurement says
Off-loading does not delete the cost, it relocates it. Three things stay on the loop:
| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |
Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.
Two results shaped the design:
- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.
## Why exactly one worker
It is load-bearing three times over, which is also why it must not quietly become a pool:
- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.
Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.
## Making every protection safe off-thread
The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:
| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |
Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.
With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".
## Correctness
- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.
## Bounding
`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.
The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.
## Gate
Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.
## Engine change
One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.
## Docs
`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.
## Testing
698 UOContent tests, 810 Server tests, Release build clean.
Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
## The bug
Any property getter reached from `GetProperties` that calls `InvalidateProperties` takes the tooltip build down with it:
```
System.ArgumentNullException: Value cannot be null. (Parameter 'array')
at Server.ObjectPropertyList.AppendStringDirect(String value)
at Server.Mobiles.PlayerMobile.GetProperties(IPropertyList list)
```
`InvalidateProperties` rebuilds **in place** — `Reset()`, then `GetProperties()` again on the same instance — and `Reset()` does two destructive things to a build already in flight:
1. **It returns the pooled interpolation buffer.** The compiler rents it in the handler ctor and returns it in the closing `Add`, so *every hole is evaluated while it is live*:
```csharp
var handler = new InterpolatedStringHandler(1, 2, list); // InitializeInterpolation() RENTS
handler.AppendFormatted(pl.Rank.Title); // <-- getter runs HERE
handler.AppendLiteral("\t");
handler.AppendFormatted(faction.Definition.PropName);
list.Add(1060776, ref handler); // consumes span, RETURNS
```
```
GetProperties(list)
├─ InitializeInterpolation() -> _arrayToReturnToPool = Rent(256) buffer LIVE
├─ « hole 1: pl.Rank.Title »
│ └─ PlayerState.Rank.get (lazy recompute)
│ └─ Invalidate() -> InvalidateProperties() -> m_PropertyList.Reset()
│ └─ Dispose(): Return(buf); _arrayToReturnToPool = null buffer GONE
└─ handler.AppendFormatted("Knight")
└─ _arrayToReturnToPool.AsSpan(_pos..)
└─ ArgumentNullException (Parameter 'array')
```
It surfaces as `ArgumentNullException` rather than `NullReferenceException` because the `Range` overload of `AsSpan` must read `array.Length`, so the BCL null-checks and names the parameter `array`.
2. **It rewinds the packet cursor**, so properties already written are overwritten by the nested pass — a silently corrupted tooltip even where the buffer survives.
## The fix: refuse, don't recover
There is no correct recovery, and retrying the build would only hide the defect. A nested invalidation now logs an error with a stack trace, **throws in `DEBUG`** so it gets found and fixed, and in `RELEASE` returns without touching the list — a possibly stale tooltip, but no crash, no corrupted packet, and nothing leaked back to the pool. Getters that genuinely must invalidate should defer:
```csharp
Timer.DelayCall(InvalidateProperties);
```
The guard flag lives on the `ObjectPropertyList`, not the entity: it is that list's own lifecycle, it costs nothing (both `Item` and `ObjectPropertyList` absorb it in existing padding, and the list is allocated lazily), and it stays correct when builds for different entities nest.
Base instance sizes are unchanged from `main`: Item 128 B, Mobile 792 B, ObjectPropertyList 72 B, PlayerMobile 1216 B.
`PropertyList` also publishes the list into `m_PropertyList` **before** building it rather than assigning through `??=` afterwards, so a nested `InvalidateProperties` sees the build in progress instead of recursing into a second throwaway list whose work is discarded.
`ObjectPropertyList` re-rents its scratch buffer instead of spanning a null array, so a stray `Reset()` from any other caller degrades rather than aborting `GetProperties`.
## Factions `PlayerState`: maintained, not lazily computed
The getter that surfaced this is now a plain field read — the whole `if (m_InvalidateRank)` block and the flag itself are gone:
```csharp
public RankDefinition Rank => m_Rank;
```
`UpdateRank()` recomputes at each point an input actually changes:
| Site | Why |
|---|---|
| `RankIndex` setter | this player's index changed |
| end of `KillPoints` setter | two paths write `m_RankIndex` directly, bypassing the setter; runs once the swap bookkeeping and `ZeroRankOffset` have settled |
| `Faction.AddMember` | *after* the insert — the member count is not settled during the ctor |
| `FactionState` load | once ordering and `ZeroRankOffset` are final |
Supporting fixes this forced out:
- **Both ctors seed the lowest rank.** Nothing recomputes on read any more, so `Rank` has to be usable immediately — including for members that never get a `RankIndex` assigned, which is *every member with no kill points*. Without this, `Rank.Title` NREs.
- **`Rank` always resolves.** Ranks are ordered by `Required` descending ending at `0`, so a *negative* percent (`RankIndex` out of sync with `ZeroRankOffset`) matched nothing and left `m_Rank` null. It no longer divides by a zero `ZeroRankOffset` either.
- **A pre-existing staleness bug.** The `KillPoints` setter writes `m_RankIndex` directly in two places, so the cached rank was never refreshed when a player crossed zero kill points.
All six readers of `Rank` were checked; none relied on the old side effect.
One behaviour change worth flagging: rank refreshes are now **eager** where they used to be lazy, so a `KillPoints` change invalidates each swapped player as it happens. The swap loops break as soon as ordering is satisfied — typically 0–2 swaps — but it is on the path that runs on every faction kill.
## Documentation
The rule is written down so it is enforceable rather than folklore:
- **CLAUDE.md** audit rule 19
- **`dev-docs/property-lists.md`** — new "Never Invalidate From Inside `GetProperties`" section with the failing/passing pattern
- **`dev-docs/claude-skills/modernuo-property-lists.md`** — key rule + anti-pattern
- **`dev-docs/claude-skills/modernuo-code-audit.md`** — rule 19, ERROR severity
## Tests
- `ObjectPropertyListReentrancyTests` — `Reset()` and `Dispose()` re-entered mid-hole (both red against `main` with the exact exception above), nesting behaviour, and the new contract: `DEBUG` throws, `RELEASE` survives, and the build is never retried into a loop.
- `FactionRankTests` — `Rank` is populated before anything reads it, tracks `RankIndex` without a read, is stable across reads, and still resolves when `RankIndex` is out of sync with `ZeroRankOffset`. Red-verified: removing the ctor seed fails the first one.
793/793 `Server.Tests` and 608/608 `UOContent.Tests` pass.
## Noted, not addressed here
`~ObjectPropertyList()` returns the rented array to `STArrayPool<char>.Shared` from the **finalizer thread**, and that pool is single-threaded by design. Left alone as a separate concern.
## At a glance
```csharp
public override void GetProperties(IPropertyList list)
{
base.GetProperties(list);
// Accumulate any number of free-text lines. On dispose the block flushes via
// AddChunked, splitting across as many OPL properties as needed so none can
// overflow the legacy 2D client's per-property buffer (which would crash it).
using var block = list.TextBlock();
if (luck > 0)
{
block.Add($"Luck Bonus: +{luck}%"); // zero-alloc interpolation
}
block.Add("Cannot be repaired".AsSpan()); // plain text, no string allocation
// Already holding a '\n'-joined string? Skip the builder and chunk directly:
// list.AddChunked(description);
}
```
## What
Adds a safe path for emitting **variable-length, free-form (non-cliloc) tooltip text**:
- **`ObjectPropertyList.Add(ReadOnlySpan<char>)`** overloads — append raw text with no string allocation. Makes the old single-arg `Add(string)` redundant (a `string` binds to the span overload implicitly), so it's dropped.
- **`AddChunked(ReadOnlySpan<char>)`** on `IPropertyList` — splits newline-joined text at `\n` boundaries across as many passthrough-cliloc properties as needed, so no single property exceeds the cap.
- **`OplTextBlock`** (`ref struct`) + **`IPropertyList.TextBlock()`** — an ergonomic builder that accumulates `\n`-joined lines (with a zero-alloc interpolated `Add($"...")` overload) and flushes via `AddChunked` on dispose. Usage: `using var block = list.TextBlock();`.
- **`MaxArgumentLength` (504)** — per-property cap with a hard backstop that clamps + logs anything that slips through.
## Why
The legacy 2D client copies each OPL property's text into a fixed ~512-char (1024-byte) buffer. A single property longer than that smashes an adjacent world object's vtable on the client heap and crashes the client. `AddChunked`/`OplTextBlock` keep multi-line content safely under the cap instead of risking one oversized `Add`.
## Docs
- `dev-docs/property-lists.md` — new "Multi-Line Free Text" deep-dive section; corrected the stale `IPropertyList` listing.
- `dev-docs/claude-skills/modernuo-property-lists.md` — condensed pattern + anti-pattern.
## Tests
9 tests pass (`OplTextBlockTests`, `ObjectPropertyListSpanAddTests`): line joining, empty-line skipping, no-line no-op, zero-alloc interpolation, and long-content chunking staying under `MaxArgumentLength`. Full `UOContent` build is green, confirming dropping `Add(string)` breaks no call sites.
## Summary
Captures the durable learnings from the message-interpolation work (PRs #2434, #2436, #2437, #2438, #2440) as reference documentation. **Doc-only PR — no code changes.**
The original Phase 2 audit (PR #2435) was development scaffolding and was closed unmerged once Phase 3 consumed it. This PR replaces it with proper reference docs that future authors can consult.
## What's added
### `dev-docs/string-handling.md`
- Promote `RawInterpolatedStringHandler` from a one-line note to a proper section listing all APIs that accept it (messages, OPL, gumps, packets).
- Document the `:L` lowercase format specifier.
- New comprehensive **"Interpolation Anti-Patterns"** section covering 8 patterns with before/after examples — applies to any handler-aware API:
1. Ternary with interpolated branches
2. Switch expression with interpolated arms
3. Pre-built local typed as `string`
4. `.ToString()` (or any string-returning method) inside a hole
5. String concatenation inside a hole
6. `string.Format` feeding a handler-aware API
7. LINQ-built strings inside a hole
8. Pre-built concat var
### `dev-docs/networking-packets.md`
- Add **"Player-Facing Message APIs"** section listing `Mobile` / `Item` / `NetState` message methods with their handler overloads.
- Note the `IBroadcastFilter` pattern for new spatial-broadcast helpers.
### `dev-docs/property-lists.md`, `dev-docs/gump-system.md`
- Cross-reference the new anti-patterns section.
- Add explicit `.ToString()` inside holes warning to property-lists (it had no such guidance before).
### `dev-docs/claude-skills/`
- Mirror the same content (condensed) in `modernuo-string-handling.md`, `modernuo-networking.md`, `modernuo-property-lists.md`, `modernuo-gump-system.md`.
- Add audit rule #17 to `modernuo-code-audit.md` covering all 8 anti-patterns with severity WARNING, plus the `:L` format spec.
### `CLAUDE.md`
- Add audit rule #18 summarizing the interpolation anti-patterns + `:L`, pointing to `dev-docs/string-handling.md` for details.
## Why this matters
Before this PR there was no documentation explaining when an interpolated string call site silently allocates a string despite the receiving API providing a handler overload. The Phase 3 cleanup (PRs #2436/#2437/#2438) discovered ~28 such sites in the codebase; without these docs the same patterns would re-emerge. The new audit rule + CLAUDE.md entry will catch them at write time.
BaseCreatures are deleted on death (Mobile.OnDeath calls Delete for non-players), so after save/restart the corpse's _owner reference resolves to null. CorpseNotoriety gated its entire creature branch on `target.Owner is BaseCreature`, falling through to player-corpse logic once the reference vanished. That made monster corpses turn red (body.IsMonster -> Murderer) and innocent NPC corpses turn grey (null is not PlayerMobile -> CanBeAttacked) on the next restart.
Snapshots the relevant owner state into CorpseFlag at corpse creation: OwnerWasBaseCreature, OwnerWasSummoned, OwnerWasAnimatedDead. Folds the standalone _murderer bool into CorpseFlag.Murderer for consistency with Criminal. CorpseNotoriety now consults the flags so the creature branch stays correct without a live mobile reference.
Bumps Corpse serialization to v16 with a MigrateFrom(V15Content) that maps the old Murderer bool onto the new flag. Pre-fix corpses already on disk decay within 7 minutes; their first post-restart color may be wrong, which is acceptable.
Also documents that the schema generator must be run after every version bump (`dotnet tool run ModernUOSchemaGenerator -- ModernUO.slnx`) since `dotnet build` does not emit migration JSON files.
## Summary
- **Add a self-referencing `InterpolationHandler` to `ValueStringBuilder`** that writes directly into the builder's buffer — zero intermediate allocation, works with `stackalloc`-backed builders
- **Replace all `System.Text.StringBuilder` usage** across the codebase with `ValueStringBuilder`
- **Convert `ValueStringBuilder.Create()` to `stackalloc`** at 10 sites where output length is provably bounded
- **Convert manual `Dispose()` to `using var`** where possible, and hoist loop-scoped builders outside loops with `Reset()`
- **Convert verbose `Append()` chains to `Append($"...")`** interpolation for readability
- **Add comprehensive documentation** for string handling patterns
## InterpolationHandler Design
`ValueStringBuilder` is a `ref struct`, which creates challenges for C#'s interpolated string handler pattern:
- **`ref` fields to ref structs are not allowed** (CS9050)
- **`[InterpolatedStringHandlerArgument("")]` passes struct receivers by value**, not by ref
- **`ISelfInterpolatedStringHandler` requires boxing** ref structs into interface fields
**Solution: Copy-and-reconcile pattern.** The handler receives a value copy of the builder. The copy shares the same underlying `char` buffer (`Span` points to the same `stackalloc`/pooled memory), so writes go to the original buffer. `Append()` reconciles by `this = handler._builder`, updating `_length` and any buffer references changed by `Grow()`.
This is safe because:
- The game loop is single-threaded — no concurrent access between handler construction and reconciliation
- If `Grow()` occurs in the copy, the original's stale buffer isn't accessed until `Append()` replaces it
- `Dispose()` correctly returns the reconciled buffer to the pool
## Changes by Category
### ValueStringBuilder (`Projects/Server/Buffers/ValueStringBuilder.cs`)
- Added nested `InterpolationHandler` ref struct with copy-and-reconcile pattern
- Added `Append([InterpolatedStringHandlerArgument("")] scoped ref InterpolationHandler)` method
- Removed `RawInterpolatedStringHandler` overloads (new handler replaces them)
- All `AppendFormatted` overloads delegate to existing `Append` methods (no code duplication)
- Alignment support via direct private field access (nested type privilege)
### StringBuilder → ValueStringBuilder (15 files)
Replaced all `new StringBuilder()` with `ValueStringBuilder.Create()` or `stackalloc`:
- ConPVP games: KingOfTheHill, DoubleDom, CTF, BombingRun, TourneyMatch
- ConPVP infrastructure: Tournament, Participant, TourneyParticipant
- ConPVP gumps: ArenaGump, TournamentBracketGump, AcceptTeamGump, ConfirmSignupGump
- Commands: Handlers, Logging, Add
- Other: TownCrier, SpeechLogGump, TestCenter
Key patterns:
- `sb = new StringBuilder()` reassignment → `sb.Reset()`
- `sb.AppendFormat("{0:N0}", value)` → `sb.Append($"{value:N0}")`
- `sb.Append(x).Append(y)` chains → separate statements (VSB returns void)
### Create() → stackalloc (10 files)
Converted heap-allocated builders to stackalloc where output is bounded:
- ClientVersion (32), MapSelection (160), HouseRaffleStone (48)
- HolySense (96), UnholySense (96), ClientVerification (192)
- AcceptTeamGump (64), ConfirmSignupGump (64)
- BaseWeapon (160), BaseArmor (128)
### Loop optimizations (2 files)
Hoisted `ValueStringBuilder` creation outside loops with `Reset()` per iteration:
- TourneyMatch.cs: `using var` inside for loop → stackalloc before loop
- ArenaGump.cs: `Create()` + `Dispose()` per iteration → stackalloc before loop
### Append chain → interpolation (5 files)
Converted multi-line `Append()` chains to `Append($"...")`:
- BountyMessage.cs: title switch (6 cases), paragraph (15→1 Append), description lines, closing
- AcceptTeamGump, ConfirmSignupGump, TournamentBracketGump: tournament type strings
- AdminGump: comment/tag formatting in loops
### Documentation
- `dev-docs/string-handling.md`: Full reference — construction, interpolation, disposal, decision guide
- `dev-docs/claude-skills/modernuo-string-handling.md`: Claude skill with quick reference
- `CLAUDE.md`: Added rule 17 (no StringBuilder), dev-docs table entry, skills table entry
- `dev-docs/code-standards.md`: Updated memory management section
## Test Plan
- [x] `dotnet build` — 0 errors, 0 warnings
- [x] `dotnet test` — 940/940 tests pass
- [x] 28 ValueStringBuilder tests covering all reconciliation scenarios:
- Stackalloc no-grow, stackalloc with grow (→pool transition)
- Heap no-grow, heap with grow, heap double grow
- Pre-existing content with and without grow
- Sequential multiple `Append($"...")` calls
- Mixed plain + interpolated Append
- Empty interpolation, literal-only, format specifiers
- Null string holes, ISpanFormattable types
- Dispose after stackalloc→pool grow
Summary
- Adds CLAUDE.md at repo root with 14 terse code audit rules (always loaded, low token cost)
- Adds pointer files for other AI tools: AGENTS.md (Codex), GEMINI.md, .github/COPILOT-INSTRUCTIONS.md (Copilot), .cursorrules (Cursor) — all redirect to CLAUDE.md as single source of truth
- Gitignores /.claude so personal AI config isn't distributed
- Moves Claude skills to dev-docs/claude-skills/ (opt-in, not auto-loaded)
- Adds 14 dev-docs covering codebase conventions
Code Audit Rules (in CLAUDE.md)
1. LINQ tiered rules (Tier 1 free, Tier 2 warm, Tier 3 forbidden)
2. No Console.WriteLine — use LogFactory.GetLogger()
3. No concurrency primitives in game code
4. No World.Mobiles/World.Items iteration
5. Clean up refs in OnDelete()/OnAfterDelete()
6. Cancel timers in OnDelete()/OnAfterDelete()
7. STArrayPool<T>.Shared not ArrayPool<T>.Shared
8. PooledRefList<T> not new List<T>() on hot paths
9. Serialization: partial class, [Constructible], no serialized TimerExecutionToken
10. No Task.Run/new Thread() in game code
11. Never assume era — ask which expansion
12. _camelCase fields, PascalCase properties/methods
13. No empty gumps — use DisplayTo() pattern
14. PropertyList string literals must be {} holes, cliloc-as-argument uses :#
Kamron Batman