## Problem
`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.
## Result
Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:
| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |
**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.
## How
The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.
**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.
**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.
## Configuration
| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |
## Diagnostics (compiled out by default)
`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.
## Verification
- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.
## Notes
The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
## Why
An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.
## What the measurement says
Off-loading does not delete the cost, it relocates it. Three things stay on the loop:
| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |
Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.
Two results shaped the design:
- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.
## Why exactly one worker
It is load-bearing three times over, which is also why it must not quietly become a pool:
- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.
Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.
## Making every protection safe off-thread
The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:
| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |
Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.
With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".
## Correctness
- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.
## Bounding
`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.
The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.
## Gate
Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.
## Engine change
One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.
## Docs
`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.
## Testing
698 UOContent tests, 810 Server tests, Release build clean.
Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
## What
`Dictionary<K,V>.Remove` and `HashSet<T>.Remove` do not bump the collection's version, so removing an entry during a `foreach` does not invalidate the enumerator. A number of loops were still paying for a `PooledRefQueue`/`PooledRefList` to collect keys and drain them in a second pass. This drops those guards.
## Why it's safe
Verified against .NET 10.0.10 rather than taken on trust, since the documented guarantee covers only `Dictionary<TKey,TValue>.Remove` while several of these call sites are `HashSet<T>` or enumerate `.Keys`/`.Values`:
| Case | Result |
|---|---|
| `Dictionary` foreach + `Remove` | safe, all entries visited |
| `Dictionary.Keys` / `.Values` foreach + `Remove` | safe, all entries visited |
| `HashSet` foreach + `Remove` | safe, all entries visited |
| `Dictionary` foreach + `Remove` **then `Add`** | throws `InvalidOperationException` |
Reflection on `_version` confirms the mechanism: neither `Dictionary.Remove` nor `HashSet.Remove` touches it. Because `Remove` never bumps the version, the `Keys` and `Values` enumerators are just as safe as the dictionary's own, even though only `Dictionary.Remove` documents the behaviour. No entries were skipped in any case.
The `HashSet` half is confirmed by [stephentoub on dotnet/dotnet-api-docs#8177](https://github.com/dotnet/dotnet-api-docs/issues/8177#issuecomment-1167251052): *"Both HashSet and Dictionary have been improved to support removal during enumeration. The docs may just benefit from updating."* The gap is in the documentation, not the runtime.
`Remove` followed by `Add` in the same enumeration still throws. That is the line this PR does not cross.
## Guards removed
`VisibilityList`, `ChampionTitleSystem`, `Channel`, `BombingRun`, `Ruleset`, `PuzzleChest`, `RaceChangeGump`, `StepCache`, `PlayerMurderSystem`, `VirtueSystem`, `ProjectedItem`, `StaminaSystem`, `AIGroupMovement`, `PromotedGuard`, `AutoDenylist`, `LoginAllowlist`, `AntiMacroSystem`, `DetectHidden`.
Both collection kinds are covered: `Dictionary` (including loops over `.Keys` and `.Values`) and `HashSet` (`ProjectedItem._active`, `PlayerMurderSystem._contextTerms`, `StaminaSystem._resetHash`). In `StaminaSystem.ResetTimer` the `Count == queue.Count → Clear()` branch goes away with the queue — it only existed to avoid paying for N individual removes.
Where the collection supports it, `Contains` + `Remove` and `TryGetValue` + `Remove` also collapse into a single lookup (`if (list.Remove(x))`, `if (m_Pending.Remove(ns, out var state))`).
`Utility.Tidy<K,V>` keeps its two branches: when `K` is serializable the value is not inspected, otherwise the value is. Only the serializable side may be cast, so `Dictionary<Mobile, int>` and `Dictionary<Mobile, string>` stay valid.
## Deliberately unchanged
**`BaseCreature.LoyaltyTimer.OnTick`** keeps its deferred-delete queue. Removing from `World.Mobiles` while enumerating it is safe, but `Mobile.Delete()` is not a `Remove` — it runs `OnDelete`/`OnAfterDelete`, the `OnParentDeleted` cascade over the creature's pack, `DropHolding()`, and region and guild callbacks. Anything in that surface that constructs a `Mobile` is an `Add` into the dictionary being enumerated, which does invalidate it. `BaseHire.PayTimer.OnTick` has the same shape and is likewise untouched.
**Spatial-query buffers** — `GuardedRegion.CallGuards`, `Thunderstorm`, `Exorcism`, `LeverPuzzleController`, `BaseCreature.TeleportPets` — are a different hazard. They buffer the result of a range query because the drain moves or harms mobiles, which mutates sectors mid-enumeration.
**Re-entrant drains.** The `_users` sets in `Firebomb` and the explosion, conflagration and confusion-blast potions look like this pattern but are not: the loop collects, `Clear()`s, and only then runs `Target.Cancel` on each, which can re-enter. `AnimalTrainer` enumerates `pm.Stabled` and drains through `RemoveStabled`, which nulls the `Stabled` field once it empties — safe for an in-flight enumerator, which holds the set reference rather than the field, but subtle enough not to be worth inlining on a cold path.
## Verification
`dotnet build` clean with 0 warnings; 810 Server and 684 UOContent tests pass.
## What
- Bind the login auth id to the account **and** origin address that earned it, make it a CSPRNG draw, expire it after two minutes, and spend it only once its owner presents it.
- Skip the password verify on `GameLogin` (0x91) when the presented id vouches for the submitted username and address.
## Why
A full client login hashes the password twice — `AccountLogin` (0x80) and then `GameLogin` (0x91). At the current Argon2 parameters that is **most of a 16 ms frame each, on the single-threaded game loop**, for every login attempt.
The second verify is redundant. `GameLogin` already requires an id from `_authIDWindow`, and that window is only populated by `GenerateAuthID`, called from `PlayServer` — reachable only after 0x80 has already authenticated the account **in this same process**. ModernUO Gateway has its own auth-id passing mechanism and is out of scope here.
## Why the id needed hardening first
Skipping the verify promotes the id from a correlation token to a bearer token, and it was not one:
- drawn from `Utility.Random` → `BuiltInRng`, a non-cryptographic PRNG
- bound to nothing — `AuthIDPersistence` carried only `Age` and `Version`
- never expiring; `Age` was only read to pick an eviction victim
A guessed id got you nothing while the password was still checked. Without that check it would have been an account takeover, so the id is now a CSPRNG draw, single-use, two-minute TTL, and bound to both the account and the origin address.
What remains is observing a live id on the client's network or machine — which the server cannot defend against under any design, and which already yields the password itself, since the client transmits it in the same handshake.
Network switching mid-login is deliberately unsupported.
## Behaviour
A full verify was always required before this change, and ids never expired, so every "before" is a password check.
| Case | Before | After |
|---|---|---|
| Id absent | Disconnect | Disconnect |
| Address mismatch | Verify | **Disconnect** |
| Account mismatch | Verify | **Disconnect** |
| Expired | Verify | **Verify** |
| Id vouches | Verify | **Skip** |
No case grants access the previous code would have denied. Expiry deliberately falls back to the verify rather than disconnecting — a player can idle, and turning that into a lockout would be a regression for no gain.
## Look, then take
An id is not consumed until the presenter has shown it is theirs. Removing it first would let anyone who lands on a live id burn it, and its owner would arrive to `"Unable to find auth id."` and have to log in again over a packet they had no part in.
The **address is compared before the account**, so a guesser from anywhere else is rejected before a username is ever looked at. That is what makes it safe to leave the id in place on a mismatch: there is no username-enumeration risk to trade against, and the only presenter who could enumerate is already on the victim's own address.
## The window is not a cap
It was 128 entries with the oldest evicted to make room. That is a cap on *concurrent logins*, not a resource bound: 800 people picking a server at once would have live ids discarded and those clients would arrive to `"Unable to find auth id."` — a failed login caused by nothing except other people logging in.
Issuing now sweeps expired entries and lets the window grow if everything in it is still live. Unbounded is safe here: an entry costs a **successful** password verify to create and dies after two minutes, so its size tracks logins genuinely in flight.
Removing an id when its connection drops is not an option, and this was checked rather than assumed — `NetState.cs:787` disconnects the login connection *deliberately*, immediately after the id is issued, and that disconnect is never cancelled. Surviving it is the whole purpose of the id. Expiry is the only correct reclamation.
## Handshake hardening
Choosing a server queues a disconnect, but the queue drains on the *next* slice, so a client pipelining into the same recv buffer can reach the handshake handlers again. Two had no do-once guard:
- `LoginServerSeed` (0xEF) now rejects when `state.Seeded` is already set.
- `PlayServer` (0xA0) now rejects when `state.AuthId != 0` — otherwise a connection that had already spent its id would be handed the spent one back.
Issuing is also idempotent (`EnsureAuthId`), so a connection holds exactly one id by construction and an orphan is impossible rather than something to clean up. The login state machine itself is untouched.
Also fixes a fall-through: the "Unable to find auth id" branch disconnected without returning, then continued with a default entry and nulled `state.Version`.
## Testing
`ConsumeAuthId` is a seam with no `NetState` dependency, so the auth decision is tested directly: vouching, account mismatch, address mismatch, case-insensitive usernames, IPv4-mapped-IPv6, unknown ids, single-use by the owner, **a rejected attempt leaving the id redeemable**, expiry-into-verify, and an 800-id login rush that must evict nobody. Expiry is driven by moving `Core._now`, not by waiting. Every new clause was verified to discriminate by removing it and confirming only its own tests fail.
## Cost
Halves the per-login game-loop cost. This does not make hashing cheaper or move it off the loop — that is gated on a measurement described in `docs/handoffs/2026-08-07-off-loop-argon2-hashing.md`.
> ⚠️ **Rollback hazard — one-way door once logins are taken.** Serialization is unchanged, so a save
> written by this build still *loads* on the previous one. Its contents do not survive the trip: on
> its first successful login each account is rehashed to `$argon2id$`, and the previous build ships
> Argon2.Bindings 1.19.0, whose `Verify` is gated by the verifier's own configured type and answers
> `false` for an `$argon2id$` hash. **After a shard running this build has accepted logins, do not
> roll back past this commit** — every account that logged in is locked out on the older binary, and
> the only recovery is rolling forward again or resetting passwords by hand. Roll back only from a
> save taken before the first post-deploy login.
Requires [Argon2.Bindings 1.20.0](https://github.com/modernuo/Argon2.Bindings/pull/14), now published.
## What
- Consume `Argon2.Bindings` 1.20.0, which resolves the Argon2 type from the stored PHC string rather than from the verifier's own configuration.
- Default to **Argon2id, m=16384, t=1, p=1** — 8.51 ms against the old Argon2i 8 MiB t=3 at 10.11 ms. Cheaper *and* stronger.
- Rehash on a successful login whenever the stored parameters are stale, not only when the algorithm changes.
- Fix `SetPassword`, which derived the password phrase from the outgoing algorithm while storing it under the incoming one.
## Why
**Verification was gated by the verifier's configured type.** `Verify` passed the instance's own `ArgonType` to native `argon2_verify`, whose `decode_string` rejects a disagreeing `$argon2i$`/`$argon2id$` prefix and returns `DECODING_FAIL` — folded into `false`, the same answer as a wrong password. Switching the default type would have locked out every existing account, and `VerifyAndUpdate` could not have migrated them either: it delegates to the same type-fixed `Verify` and never compared `ArgonType`. Fixed upstream in 1.20.0. The pinned legacy-`$argon2i$` test here fails on 1.19.0 for exactly that reason, which is what makes the package bump load-bearing rather than incidental.
**Changing the defaults would otherwise have reached nobody.** Argon2's PHC string embeds `m`, `t` and `p`, so verification uses the parameters stored with each account, not the configured ones — and verification is the hot path. `CheckPassword` only rehashed when the *algorithm* changed, never when its cost parameters did, so on an established shard the new defaults would have applied to new accounts only. `IPasswordProtection.NeedsRehash` closes that: it defaults to `false`, so PBKDF2 and the `HashAlgorithm` protections are untouched — only Argon2 carries its cost inside the stored value.
**`SetPassword` picked the phrase rule from the wrong algorithm.** SHA1 and SHA2 salt the phrase with the username; Argon2 and PBKDF2 do not. It chose the rule from the *outgoing* algorithm while storing under the *incoming* one, so any algorithm change wrote a credential its own next verify could not reproduce. It now assigns `PasswordAlgorithm` first and derives the phrase from that. Note this ordering is load-bearing and invisible — `UpgradingAlgorithm_DoesNotLockTheAccountOut` is what pins it.
## Cost
Verification is re-derivation, so these are login numbers. A full login calls `CheckPassword` twice — `AccountLogin` (0x80) then `GameLogin` (0x91): **~20 ms before, ~17 ms after**, plus a one-time ~8.5 ms rehash on each account's migrating login.
That cost is still paid on the game loop. Moving hashing off-loop is deliberately **not** in this PR — it needs a pending-auth state in the login handlers, bounding of in-flight hashes, and login rate limiting.
## Why
ModernUO mandated `-dev` packages on production servers for exactly one reason: `DllImport` never
asks for a versioned SONAME, so `libdeflate.so.0` and `libargon2.so.1` sitting in `/usr/lib` went
unfound, and the `-dev` package's unversioned symlink was the only thing making resolution work.
The `-dev` packages ship no library of their own — operators were installing headers and a static
lib on machines that compile nothing.
Fixed in the binding packages (modernuo/LibDeflate.Bindings#4, modernuo/Argon2.Bindings#13), so
this picks them up and stops asking.
```
LibDeflate.Bindings 1.0.3 -> 1.0.4
Argon2.Bindings 1.17.0 -> 1.19.0
```
## zstd is dropped too, on every platform
ZstdNet bundles `libzstd` for `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64` and win, and
nothing shells out to the CLI. Verified: the 15 `ManagedArchive` round-trip tests pass in a
container with no `zstd` package installed and `which zstd` empty. Removed from the README, the
macOS `brew install`, and CI — so the macOS runners now prove it rather than us assuming it.
## NativeLibraryChecker asks a different question
It asked *"is package X installed"* via `dpkg -l` / `rpm -q`. That is what forced `-dev`, and no
hardcoded name works for ICU anyway — its apt package is release-specific (`libicu70` on Ubuntu
22.04, `libicu76` on Debian 13). It now asks *"will the loader find this"*: `NativeLibrary.TryLoad`
on the unversioned name, then `libfoo.so.N` descending through the range the runtime accepts.
It deliberately does not consult a package database or `ldconfig -p`. Both answer a different
question than "will `dlopen` succeed" — see the ICU section below for how that bit.
## What was wrong with the ICU check
`libicuuc` was **inherited, not derived**. It came from translating the old package-name check into
a library probe, without establishing which library that should be. Reviewing it turned up three
defects, all of which could report ICU present on a host where the runtime then refuses to start:
- **`libicui18n` was never probed.** The only ICU names in `libSystem.Globalization.Native.so` are
`libicuuc` and `libicui18n`. `libicudata` arrives as a dependency of `libicuuc`, and
`libicuio`/`libicutu`/`libicutest` are never referenced — so that is the complete list, and both
are checked now.
- **No version floor.** The runtime's `MinICUVersion` is 60, but the probe accepted down to
`.so.0`. RHEL/CentOS 7 ships ICU 50, which passed and then aborted at startup.
- **The `ldconfig` fast path bypassed the range.** A cache line for `libicuuc.so.50` still matches a
`libicuuc.so` prefix test, so the floor was unenforceable through it. It also trusts a stale
cache — observed reporting a deleted `libdeflate` as present. Removed in favour of asking the
loader directly, which reads the same cache but answers the real question, and which also deletes
the musl special-case (`ldconfig -p` exits 0 on musl while producing nothing usable).
Worth knowing when this goes wrong in the field: **missing ICU does not throw, it `FailFast`s** —
SIGABRT, exit 134, uncatchable. The process starts cleanly and dies later at whatever line first
touches a culture, so the stack rarely implicates ICU.
## tzdata is a separate prerequisite, and nothing was checking it
The event scheduler resolves configured zone IDs through `TimeZoneInfo`, which reads
`/usr/share/zoneinfo`. It is data rather than a library, so no loader probe finds it, and slim
container images routinely omit it. Without it every lookup except `UTC` throws
`TimeZoneNotFoundException` and `GetSystemTimeZones()` returns 1 entry instead of ~419.
There is no per-zone packaging to opt into — it is ~2 MB for the whole set. The one split that does
exist is a trap rather than an optimization: Debian 12 and Ubuntu 24.04 move the legacy aliases into
`tzdata-legacy`, so plain `tzdata` has `America/New_York` and `EST5EDT` but is **missing
`US/Eastern` and `Asia/Calcutta`**. A shard configured with a legacy alias throws even though tzdata
is installed. Documented, with both fixes.
## Why `InvariantGlobalization` stays false
Dropping ICU entirely by turning on invariant mode looks tempting and is not safe. Because
`Directory.Build.props` also sets `PredefinedCulturesOnly=false`, invariant mode does **not** throw
`CultureNotFoundException` — it silently hands back invariant data. Measured on .NET 10:
| Behaviour | With ICU | Invariant mode |
|---|---|---|
| `new CultureInfo("de-DE")` | real culture | succeeds, returns invariant data |
| de-DE decimal separator | `,` | `.` |
| `1234.5` as de-DE | `1.234,5` | `1,234.5` |
| `string.Compare("a", "B", InvariantCulture)` | `-1` (linguistic) | `31` (ordinal) |
| sort `[b, A, a, B]` | `a, A, b, B` | `A, B, a, b` |
| `FindSystemTimeZoneById("Eastern Standard Time")` on Linux | resolves | `TimeZoneNotFoundException` |
| UTF-8 round-trip of non-ASCII | unaffected | unaffected |
Number parsing and formatting produce wrong values with no error, and culture-sensitive sort order
silently becomes ordinal. Encoding is not the mechanism — UTF-8 round-trips fine either way.
## Documentation
The rationale now lives in `dev-docs/platform-prerequisites.md` rather than in comments, so it is
discoverable without reading the build tool: what each dependency is for, what breaks without it,
per-distro package names, the ICU floor, the `tzdata-legacy` split, and why the check asks the
loader instead of the package manager.
README drops `libicu-dev`. Matching the runtime package by pattern (`'^libicu[0-9]+$'`) is
version-independent without pulling in headers, so **no `-dev` package is required on any supported
distribution** — which was the point of the whole change.
## CI now proves the claim instead of contradicting it
The dnf job already installed runtime packages only. The apt job installed `libicu-dev`, which ships
the unversioned `libicuuc.so` symlink — so every probe succeeded on the first attempt and the
versioned-SONAME fallback this PR depends on was never exercised. Switched to the pattern match,
verified to resolve exactly one package on jammy (70), bookworm (72), noble (74) and trixie (76).
Added an assertion that the unversioned symlinks are absent. Without it the suite silently stops
testing anything the moment a base image starts shipping one. Verified against all eight matrix
distributions — none ship them — and confirmed the step fails as intended when a symlink is planted.
## Audit of every other native entry point
Checked whether anything else has the same hazard. It does not:
| Import | Verdict |
|---|---|
| `ws2_32.dll` — `SocketHelper` | Always present on Windows |
| `libc` — `SocketHelper` | **Verified safe**, see below |
| ZstdNet → `libzstd` | Bundled for every RID |
| IORingGroup | No native library; raw syscalls |
| ICU | Loaded by the .NET runtime itself, which probes versioned suffixes |
`libc` deserved a hard look, because `libc.so` *is* a `libc6-dev` linker script while the real
library is `libc.so.6` — the same shape as the bug being fixed. It is not affected. Measured in a
container with no `libc6-dev`:
```
/usr/lib/x86_64-linux-gnu/libc.so ABSENT
/lib/x86_64-linux-gnu/libc.so.6 present
TryLoad("libc") LOADED <- resolves where "libdeflate" would not
TryLoad("libc.so") not found
getpid() -> DllImport("libc") WORKS
```
Confirmed on Alpine/musl as well. No code in this repo registers a `DllImportResolver`, and nothing
else P/Invokes.
## `--check-prereqs`
New flag. `Program.cs` only ran the SDK check in non-interactive mode — `NativeLibraryChecker` was
reachable only through the Spectre-driven guided flow, so there was no way to verify a deployment
target from a script or a container. It is what made the container verification below possible, and
it prints the exact ICU package for the running release via `apt-cache`.
It renders through the same `PrerequisiteChecker` the guided menu uses, rather than a second
hand-rolled table that could drift from it. Spectre drops ANSI styling on its own when stdout is not
a terminal, so redirected output stays clean; the console width is widened in that case so the
install hints, which are shell commands meant to be copied, do not gain a newline mid-command.
```
╭───────────────────────────╮
│ Checking native libraries │
╰───────────────────────────╯
✔ libicuuc (Found)
✔ libicui18n (Found)
❌ libdeflate (Not found)
❌ tzdata (Not found — every zone except UTC will throw)
⚠️ Install the missing dependencies. The -dev/-devel packages are not required:
sudo apt-get install -y libicu74 libdeflate0 tzdata
```
Exit code carries the machine-readable half: 0 when everything resolves, 1 when anything is missing.
## Verification
Against 1.0.4 and 1.19.0: build plus **810 Server.Tests and 642 UOContent.Tests**, on Windows and
on Linux with **only** `libdeflate0` and `libargon2-1` installed — with the absence of the
unversioned symlink asserted first so the run could not pass for the wrong reason.
`--check-prereqs` verified in containers on Debian and Alpine across every state that matters: all
present, each dependency removed individually, tzdata removed, a deliberately stale `ldconfig`
cache, and ICU downgraded to `.so.50` to confirm the floor rejects it. Package resolution and the
absence of unversioned symlinks checked on all eight CI distributions.
## Why
`IORingGroup` issues io_uring syscalls directly rather than linking `liburing`, so the package has
never been needed — but we ask operators to install it in the README, install it in CI, and check
for it in `build-tool`.
Verified against the **shipped** `IORingGroup` 1.0.9 assembly, not just the source:
| Symbol | Occurrences in `IORingGroup.dll` |
|---|---|
| `libc`, `libSystem.dylib`, `kernel32.dll`, `kernelbase.dll`, `ws2_32.dll` | present |
| `liburing` | **0** |
| `io_uring_queue_init` — liburing's entry point | **0** |
| `io_uring_setup` — the raw syscall | 1 |
If it linked liburing it would call `io_uring_queue_init` / `io_uring_submit`. It calls neither.
## What changes
Nine lines across three files, removing `liburing-dev` / `liburing-devel` from:
- `README.md` — both the dnf and apt prerequisite blocks
- `.github/workflows/build-test.yml` — both install steps
- `Projects/BuildTool/Prerequisites/NativeLibraryChecker.cs` — the cross-compile target text, the
apt and dnf package lists, and the `ldconfig` fallback map
Nothing else is touched. `zstd` and the `-dev` packages are a separate discussion and a separate PR.
## Risk
None to the build. `liburing` was only ever installed, never linked or loaded — removing it cannot
change resolution behaviour. `build-tool` builds clean.
This was found while investigating why Linux requires `-dev` packages at all; that fix lives in the
binding packages (modernuo/LibDeflate.Bindings#4, modernuo/Argon2.Bindings#13) and lands separately
once those publish. This piece is independent and unblocked, hence its own PR.
## Problem
`IORingGroup`'s `WindowsManagedRIOGroup.DequeueRioCompletions` stackallocs `RIORESULT[256]` (6144 bytes) and runs **once per game-loop iteration** — `NetState.Slice` → `RingSocketManager.ProcessCompletions` → `PeekCompletions` → `DequeueRioCompletions`.
The 1.0.8 package was compiled with the `.locals init` IL flag set, so every one of those calls memset the full 6 KiB before `RIODequeueCompletion` overwrote the entries it actually filled.
An EventPipe profile of a near-idle shard (3 vCPU VPS, world saves off, one player logging in and moving around) put `System.Buffer.ZeroMemoryInternal` — called directly from `DequeueRioCompletions` — at **~2.8% of main-thread samples**, and it was the dominant frame in several 60–127 ms game-loop stalls.
## Why our existing attribute didn't cover it
`Projects/Server/Module.cs` and `Projects/UOContent/Module.cs` already declare `[module: SkipLocalsInit]`. That attribute is a **compile-time** directive: it clears the flag in the IL of the assembly being compiled, and does not cross assembly boundaries. It never applied to the package.
Verified by reading the shipped IL (`MethodBodyBlock.LocalVariablesInitialized`):
| Assembly | attribute | methods with `.locals init` |
|---|---|---|
| `Server.dll` | present | 0 of 5439 |
| `IORingGroup` 1.0.8 | **absent** | **158** |
| `IORingGroup` 1.0.9 | present | **0 of 389** |
## Testing
Built and tested against the locally-built 1.0.9 package (temporary local feed, not committed):
- `dotnet build -c Release` — **0 warnings, 0 errors**
- `Server.Tests` — **810 passed, 0 failed**
- `UOContent.Tests` — **637 passed, 0 failed**
- Confirmed the `IORingGroup.dll` deployed to `Distribution/` is the fixed build (0 of 389 methods zeroing)
Only the `<PackageReference>` version changes; no source changes on this side.
## Summary
Players could not place **any door** while customizing a house, and placing other pieces could disconnect them outright. Staff saw neither problem: `HouseFoundation.Designer_Build` only enforces `ValidPiece` below `GameMaster`.
Original report and diagnosis by @SynPDX.
## Root cause 1 — no door is ever registered
The retail client's `doors.txt` separates its header rows with lines of **bare tabs** (it is the only sheet that does):
```
int<TAB>int<TAB>...<TAB>string
<TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB> <-- 10 tabs, not an empty line
Category<TAB>Piece1<TAB>...<TAB>FeatureMask<TAB>Comment
<TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB><TAB>
0<TAB>1657<TAB>1659<TAB>...
```
`Spreadsheet.ReadLine` skipped a line only when `line.Length > 0`. A 10-tab line has length 10, so it was returned as the **names row** — every column ended up named `""`, `GetColumnID("Piece1")` and friends returned `-1`, and not one of the 230 door graphics was registered. Unregistered item IDs keep the `-1` sentinel, and `CheckValidity` rejects those, so `ValidPiece` refused every door.
ClassicUO skips these lines (`string.IsNullOrWhiteSpace` in `HouseCustomizationManager.ParseFile`), which is why the client happily offers doors the server then rejects.
Measured against a retail 7.0.x `doors.txt` using the shipped `Spreadsheet`:
| | `FeatureMask` column | door graphics registered |
|---|---|---|
| before | `-1` | **0** |
| after | `9` | **230** |
## Root cause 2 — `IndexOutOfRangeException` out of the packet handler
Every sheet ends in a cosmetic `Comment` column that ModernUO never reads, and client sheets write an empty comment as a plain newline with no trailing tab. `Split('\t')` then returns one field fewer than the header declares, and the parser indexed past the end:
```
System.IndexOutOfRangeException: Index was outside the bounds of the array.
at Server.Multis.Spreadsheet..ctor(String path)
at Server.Multis.ComponentVerification.LoadSpreadsheet(...)
at Server.Multis.ComponentVerification.IsItemValid(Int32 itemID)
at Server.Multis.HouseFoundation.ValidPiece(Int32 itemID, Boolean roof)
at Server.Multis.HouseFoundation.Designer_Build(NetState state, ...)
```
The client's own parser only requires the columns up to `FeatureMask` — ClassicUO's `CustomHouseMisc.Parse` guards on `scanf.Length >= 12` for a 13-column `misc.txt` — so such a row is valid data listing real pieces. Missing trailing fields are now treated as empty rather than dropping the row, which would unregister every piece the row lists and reproduce the door symptom.
`EnsureLoaded` also set `_loaded` before loading, so once the throw escaped, an all `-1` table stayed cached and rejected everything for players from then on — the same player-visible symptom as #2500.
## Also made explicit rather than accidental
- **Named the table sentinels.** `NotAComponent` (-1) is the anti-cheat guard and the initial state; `NoFeatureRequired` (0) is a piece with no expansion gate — how `walls.txt` encodes pre-AOS base pieces and what `housing.bin` collapses to under `HousingTierMask` (#2500).
- **A sheet with no `FeatureMask` column is refused and logged.** `GetInt32` on a missing column returns 0 = `NoFeatureRequired`, which would have silently marked every piece in that sheet unconditionally placeable regardless of expansion. This was previously only harmless by accident.
- **A sheet matching none of its expected tile columns is refused and logged** — that is what `doors.txt` was doing silently. Individual missing columns stay tolerated, since older sheets predate columns such as `walls.txt`'s `SecondAltWindowS`/`E`.
- **Catch per sheet**, so one unreadable file no longer costs the other six.
- **Header guards**: an empty file or a types-only file raised a `NullReferenceException`; a names row shorter than the types row indexed past the end.
- **Fall back to the component sheets when `housing.bin` cannot be read**, instead of passing `null` into a `SpanReader`.
`_loaded` is still set before loading, deliberately: this runs from the design packet handler, and retrying would re-read every sheet on each subsequent placement attempt.
Sheet precedence is **unchanged** — the client's copies stay authoritative and `Data/Components` remains the fallback.
## Verification
- Retail 7.0.x client `doors.txt` through the shipped `Spreadsheet`: 0 door graphics before, 230 after.
- 5 new tests in `SpreadsheetTests` covering the tab separators, the omitted trailing field, per-row recovery, and both header guards. All 5 fail against `main` and pass here.
- `dotnet build` clean (0 warnings, 0 errors); `UOContent.Tests` 642/642.
## Why
The shard owner, on a Starlink CGNAT address, was blocked by the imported reputation blocklist.
The cause was not CrowdSec. The address was a literal line in `ip-blocklist.txt`, so `BlocklistFilter` denied it at accept and then promoted it — and clearing the CrowdSec decision could not fix it either, because the file entry re-reports within `promoteSuppression` of every reconnect attempt.
This is structural, not a one-off. Reputation feeds list shared consumer address space constantly: on CGNAT one public address fronts many subscribers **at the same time**, so a single abusive customer gets the address listed and everyone else behind it is blocked with them. Where leases rotate, a listing says little about whoever holds the address now. Around 1,000 Starlink addresses sit in the current list.
So exemptions go where they cost nothing, and escalation is driven by what a connection actually does.
## Generator — `tools/Export-IpBlocklist.ps1`
`-AllowlistFile` takes multiple paths, subtracted from the merged set before the output is written. Defaults to every `ip-allowlist*.txt` beside the output, merged into one allow set:
- `ip-allowlist.txt` — operator exemptions, created once and **never rewritten**
- `ip-allowlist-<name>.txt` — a carve-out you built, regenerable and copyable between shards
**Subtraction is range-correct.** An allowlisted address inside a blocked CIDR splits that CIDR around the hole rather than being silently ignored. This also fixes `-ExcludeAnonymizers`, which parsed CIDR entries into `$anonCidr` and then only ever subtracted singles.
**No carve-out ships.** A carve-out names a real network, and which ones a shard should exempt depends on where its players actually are — so publishing one would make that policy call for every shard and put a specific provider's address space in the repo. The script builds them on request instead:
```powershell
.\Export-IpBlocklist.ps1 -AddCarveout starlink -Asn 14593
```
Carve-outs are **discovered, not configured**: every `ip-allowlist*.txt` beside the output is subtracted, by the generator and by the shard, so a file an admin adds needs no config edit and no code change. Each carries an `asn=` marker in its header, which is how `-RefreshCarveouts` rebuilds it without the script keeping a list of anyone's networks; a hand-written allowlist has no marker and is never rewritten.
Prefixes come from **announcements, not ownership records**, because registry data disagrees with what is actually routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and `206.83.96.0/19` reads as APNIC in RDAP even though `206.83.96/21` is announced by Starlink.
Editing an allowlist bypasses `-MinInterval`, so a just-added exemption isn't indistinguishable from the allowlist not working. A Starlink carve-out, if you build one, costs **~4,300 IPs + ~144 CIDRs of 4.2M (0.10%)**.
## Allowlists
**`FileAllowlist`** reads the same files the generator subtracts, so an operator entry means "leave this address alone" for real. Subtraction alone only covers being *blocked*; behavioural detections never consult the blocklist, so without this a carve-out was quietly routed around — one scanner behind a shared address was enough to get everyone behind it contributed and firewalled, with nothing in the shard's own config explaining why. Reading the files also means an entry applies on the next reload rather than the next regeneration, which is what matters when someone is complaining now.
**`LoginAllowlist`** is earned by authenticating, with a 90-day TTL because an address that logged in years ago is a stranger. Its own store rather than `Account.LoginIPs`, which has no timestamps and cannot be backfilled. An entry is evidence rather than a licence: 10 suppressed contributions in an hour revokes it, and a fresh login forgives the tally.
Both are consulted **only after the blocklist has already matched**, so a normal accept pays nothing for them and the accept gate stays allowlist-free. `BanExemptions` combines them behind `BanChannel.IsExempt` and suppresses escalation only — every local defence still applies.
Two limits, both deliberate and documented in the class: `LoginAllowlist` **cannot bootstrap** (an entry is only earned by getting in, so it never repairs an existing false positive), and it is weakest on rotating CGNAT. That is why `FileAllowlist` is the fix for those, and why it is manual.
## Behavioural detection
| Reason | Trigger |
|---|---|
| `silent-connect` | Reaped after 5s having sent **zero bytes** |
| `invalid-seed` | Opened with a zero seed |
| `foreign-protocol` | Positively identified as HTTP, TLS or SSH |
**`ForeignProtocol` inverts the test.** Asking "is this a good UO client?" cannot work: `LoginEncryption.ClientDecrypt` is a byte-for-byte stream XOR, so a legitimate client with encryption enabled when the shard expects none sends a structurally perfect connection whose payload is noise. "Speaks HTTP" is safe where "unreadable" is not — however misconfigured a UO client is, it never sends `GET / HTTP/1.1`.
Nothing assumes arrival framing. TCP has no message boundaries, so a rule of the form "these bytes must arrive together" is broken by construction and drops real players on poor links. A prefix match with too few bytes to confirm waits for more. A four-byte seed can legitimately spell `GET ` (the address 71.69.84.32) or `0x16 0x03 0x0?` (22.3.x.x), so confirmation requires the request line to continue in printable ASCII or an actual ClientHello inside a plausible record — a real client's fifth byte is a packet id (`0x80`, `0x91`, `0xEF`), none of them printable, so those collisions fall through.
Everything is keyed on **bytes-received rather than elapsed time**. A connection that sent something and ran out of time is far more likely a slow link than an attack, and banning those produces the worst failure mode available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being firewalled off.
## `AutoDenylist`
A short-lived local hold (15m) on behavioural detections, as `IConnectionFilter` + `IBanReporter` over one store so the engine detection sites never reach into content.
This closes the gap where a flood pays for a socket, buffer and `NetState` slot per connection while waiting for the OS bouncer — the verdicts that matter most are reachable only *after* reading bytes — and it is the entire defence on a shard running no bouncer, which is the default config. Not persisted: a holding pen that survives restarts is a ban without a ban's review.
Cost: one dictionary lookup on a usually-empty dict per accept.
## `BanReasons`
Centralises the reason slugs. `IsBehavioral` is an **opt-in** set, not "everything except manual", so a future reason escalates normally instead of silently inheriting an exemption or entering a local denylist.
This caught a real bug during review: the first cut of the exemption swallowed `manual` admin bans (`Commands.cs`, three sites in `AdminGump`) for any allowlisted address.
## Fixes found in review
- **`BanConfiguration.Settings` was null until `Configure()` ran**, while the reap path dereferences it every `Slice()`. A harness driving `NetState.Slice()` directly hit an NRE that presented as flaky because it depended on whether an earlier test had already called `Configure()` — which is why it failed on some CI platforms and not others. Now starts at the record's defaults, with idempotency tracked by a flag; this also removes the same latent NRE from the pre-existing rate-limit path.
- **`-AllowlistFile` was typed `[string]`** while documented and used as a list, so passing two paths would have collapsed them into one string.
## Layout and docs
Content network code moves out of `Misc/` into `UOContent/Network/`, one concern per folder — `AutoDenylist/`, `Blocklist/`, `CrowdSec/`, `Firewall/`, `LoginAllowlist/`, `Packets/`. **Namespaces are untouched**, so these are pure file moves (git tracks all 16 as renames).
`dev-docs/ip-bans-and-allowlists.md` documents the subsystem, leading with the operator process for unblocking a player — including the three things that look sufficient and are not: deleting the CrowdSec decision alone, editing `ip-blocklist.txt` by hand, and `cscli allowlists` alone. `.gitignore` covers the new config files.
## Testing
Build clean. **Server.Tests 810 passed**, **UOContent.Tests 637 passed**, zero warnings. This branch adds 38 tests; the rest of the delta is main's, since this is rebased on current `main`.
New coverage: TTL boundary and renewal, private-address exclusion, manual-ban-never-exempt, unopted-reason-never-exempt, strike revocation, quiet-window reset, login forgiveness, file-allowlist CIDR coverage, file-allowlist not spending the earned list's strikes, denylist expiry-on-read, cap enforcement, lapsed-entry reclaim, HTTP/TLS/SSH identification, seed-collision fall-through, and encrypted-login-is-not-foreign.
Generator verified end-to-end against live feeds: a clean run ships no carve-out, `-AddCarveout starlink -Asn 14593` fetches and collapses 213 prefixes to 115 ranges in 0.1s over 4.2M entries, `-RefreshCarveouts` rediscovers it by its `asn=` marker, a hand-written allowlist is left untouched, and deleting a carve-out drops it rather than having it rewritten. CIDR splitting verified exhaustively: a single-IP hole in a /24 leaves exactly 255 of 256 addresses blocked.
## Operator note
Existing installs are unaffected until the generator next runs, which creates `ip-allowlist.txt` and nothing else. To unblock someone: add the address to that file and delete any live CrowdSec decision — the existing ban outlives the config change. The shard picks the entry up on its next reload, so re-running the generator is optional.
A shard whose players are on CGNAT (satellite, mobile, or an ISP short on IPv4) will likely also want `-AddCarveout`; see `dev-docs/ip-bans-and-allowlists.md`.
## Also included: a latent CI failure this PR surfaced
`fix(tests): serialize test classes that rent through STArrayPool` touches a property-list test file that has nothing to do with this feature. It is here because it was failing macOS CI, and it is trivially cherry-pickable out if you would rather it went to `main` on its own — **which may be the better call, since it is failing `main` today.**
CI has since gone green with it applied.
`STArrayPool` is single-threaded by design and its bucket cache is a plain `static`, not `[ThreadStatic]`, with a check-then-act initialize in `Return()`:
```csharp
var cacheBuckets = _cacheBuckets ?? InitializeBuckets();
```
Two threads both see null, both initialize, and the loser trips `Debug.Assert(_cacheBuckets is null)`. Anything renting from it has to stay off parallel test threads — which is what the `DisableParallelization` collections are for.
- `ObjectPropertyListReentrancyTests` and `ObjectPropertyListNestedBuildTests` (added in #2555) build property lists, which rent the interpolation buffer, but were not in the sequential collection — unlike `PropertyListInvalidationDuringBuildTests` in the same file. This is a **latent failure already on `main`**; it is timing-dependent, so it shows on some platforms and not others.
- `AutoDenylistTests` (added here) has the same exposure: its cap tests reach `AutoDenylist.Sweep`, which rents a `PooledRefList` without `mt`. The blocklist tests need no marking because `BlocklistSnapshot.Build` asks for the `mt` pool explicitly.
No production change — `STArrayPool` is the right pool on the game loop, where both `Sweep` and the property list actually run.
## Deliberately not included
Waiting for a fragmented four-byte seed at `AwaitingSeed`. It looked like a bug but the disconnect is a deliberate defence: only pre-0xEF clients reach it (0xEF goes through `HandlePacket`, which already waits for its 21 bytes), and waiting converts an instant drop into a full 5s slot hold for a client sending one or two bytes, or a loris dribbling a byte every few seconds. Against a fixed 4096-entry `MaxConnections` table that trades capacity that matters for a fragmentation case a reconnect already fixes.
## The bug
Any property getter reached from `GetProperties` that calls `InvalidateProperties` takes the tooltip build down with it:
```
System.ArgumentNullException: Value cannot be null. (Parameter 'array')
at Server.ObjectPropertyList.AppendStringDirect(String value)
at Server.Mobiles.PlayerMobile.GetProperties(IPropertyList list)
```
`InvalidateProperties` rebuilds **in place** — `Reset()`, then `GetProperties()` again on the same instance — and `Reset()` does two destructive things to a build already in flight:
1. **It returns the pooled interpolation buffer.** The compiler rents it in the handler ctor and returns it in the closing `Add`, so *every hole is evaluated while it is live*:
```csharp
var handler = new InterpolatedStringHandler(1, 2, list); // InitializeInterpolation() RENTS
handler.AppendFormatted(pl.Rank.Title); // <-- getter runs HERE
handler.AppendLiteral("\t");
handler.AppendFormatted(faction.Definition.PropName);
list.Add(1060776, ref handler); // consumes span, RETURNS
```
```
GetProperties(list)
├─ InitializeInterpolation() -> _arrayToReturnToPool = Rent(256) buffer LIVE
├─ « hole 1: pl.Rank.Title »
│ └─ PlayerState.Rank.get (lazy recompute)
│ └─ Invalidate() -> InvalidateProperties() -> m_PropertyList.Reset()
│ └─ Dispose(): Return(buf); _arrayToReturnToPool = null buffer GONE
└─ handler.AppendFormatted("Knight")
└─ _arrayToReturnToPool.AsSpan(_pos..)
└─ ArgumentNullException (Parameter 'array')
```
It surfaces as `ArgumentNullException` rather than `NullReferenceException` because the `Range` overload of `AsSpan` must read `array.Length`, so the BCL null-checks and names the parameter `array`.
2. **It rewinds the packet cursor**, so properties already written are overwritten by the nested pass — a silently corrupted tooltip even where the buffer survives.
## The fix: refuse, don't recover
There is no correct recovery, and retrying the build would only hide the defect. A nested invalidation now logs an error with a stack trace, **throws in `DEBUG`** so it gets found and fixed, and in `RELEASE` returns without touching the list — a possibly stale tooltip, but no crash, no corrupted packet, and nothing leaked back to the pool. Getters that genuinely must invalidate should defer:
```csharp
Timer.DelayCall(InvalidateProperties);
```
The guard flag lives on the `ObjectPropertyList`, not the entity: it is that list's own lifecycle, it costs nothing (both `Item` and `ObjectPropertyList` absorb it in existing padding, and the list is allocated lazily), and it stays correct when builds for different entities nest.
Base instance sizes are unchanged from `main`: Item 128 B, Mobile 792 B, ObjectPropertyList 72 B, PlayerMobile 1216 B.
`PropertyList` also publishes the list into `m_PropertyList` **before** building it rather than assigning through `??=` afterwards, so a nested `InvalidateProperties` sees the build in progress instead of recursing into a second throwaway list whose work is discarded.
`ObjectPropertyList` re-rents its scratch buffer instead of spanning a null array, so a stray `Reset()` from any other caller degrades rather than aborting `GetProperties`.
## Factions `PlayerState`: maintained, not lazily computed
The getter that surfaced this is now a plain field read — the whole `if (m_InvalidateRank)` block and the flag itself are gone:
```csharp
public RankDefinition Rank => m_Rank;
```
`UpdateRank()` recomputes at each point an input actually changes:
| Site | Why |
|---|---|
| `RankIndex` setter | this player's index changed |
| end of `KillPoints` setter | two paths write `m_RankIndex` directly, bypassing the setter; runs once the swap bookkeeping and `ZeroRankOffset` have settled |
| `Faction.AddMember` | *after* the insert — the member count is not settled during the ctor |
| `FactionState` load | once ordering and `ZeroRankOffset` are final |
Supporting fixes this forced out:
- **Both ctors seed the lowest rank.** Nothing recomputes on read any more, so `Rank` has to be usable immediately — including for members that never get a `RankIndex` assigned, which is *every member with no kill points*. Without this, `Rank.Title` NREs.
- **`Rank` always resolves.** Ranks are ordered by `Required` descending ending at `0`, so a *negative* percent (`RankIndex` out of sync with `ZeroRankOffset`) matched nothing and left `m_Rank` null. It no longer divides by a zero `ZeroRankOffset` either.
- **A pre-existing staleness bug.** The `KillPoints` setter writes `m_RankIndex` directly in two places, so the cached rank was never refreshed when a player crossed zero kill points.
All six readers of `Rank` were checked; none relied on the old side effect.
One behaviour change worth flagging: rank refreshes are now **eager** where they used to be lazy, so a `KillPoints` change invalidates each swapped player as it happens. The swap loops break as soon as ordering is satisfied — typically 0–2 swaps — but it is on the path that runs on every faction kill.
## Documentation
The rule is written down so it is enforceable rather than folklore:
- **CLAUDE.md** audit rule 19
- **`dev-docs/property-lists.md`** — new "Never Invalidate From Inside `GetProperties`" section with the failing/passing pattern
- **`dev-docs/claude-skills/modernuo-property-lists.md`** — key rule + anti-pattern
- **`dev-docs/claude-skills/modernuo-code-audit.md`** — rule 19, ERROR severity
## Tests
- `ObjectPropertyListReentrancyTests` — `Reset()` and `Dispose()` re-entered mid-hole (both red against `main` with the exact exception above), nesting behaviour, and the new contract: `DEBUG` throws, `RELEASE` survives, and the build is never retried into a loop.
- `FactionRankTests` — `Rank` is populated before anything reads it, tracks `RankIndex` without a read, is stable across reads, and still resolves when `RankIndex` is out of sync with `ZeroRankOffset`. Red-verified: removing the ctor seed fails the first one.
793/793 `Server.Tests` and 608/608 `UOContent.Tests` pass.
## Noted, not addressed here
`~ObjectPropertyList()` returns the rented array to `STArrayPool<char>.Shared` from the **finalizer thread**, and that pool is single-threaded by design. Left alone as a separate concern.
## Problem
Contributing a ban to CrowdSec failed against a real LAPI — `POST /v1/alerts` answered **500**, and depending on the shard's locale, auth answered **401**. Three independent defects, each sufficient on its own.
## Fixes
**`scenario_hash` / `scenario_version` were never serialized.** LAPI dereferences both unconditionally when persisting an alert, so omitting them is a nil deref and a 500 rather than a validation error. Both are now emitted with the values a watcher without a hub scenario is expected to send (`""` and `"1.0"`).
**`start_at`/`stop_at` were formatted without an `IFormatProvider`.** `:` is the time separator *specifier* in a custom .NET format string, not a literal — a shard running under a culture like `fi-FI` emitted `T15.04.05.123Z`, which Go's `time.RFC3339` rejects, producing another 500. Non-Gregorian cultures (`th-TH`, `ar-SA`) would also shift the year. Formatting is now pinned to `InvariantCulture` in `FormatTimestamp`, which additionally converts non-UTC input — the trailing `Z` is a literal and was previously an unchecked claim.
**The `User-Agent` was a plain product string.** LAPI's default watcher profile matches the `crowdsec/` prefix and answers 401 without it, so the header is a protocol constraint, not cosmetic. It is now an `internal const` carrying that reason.
Also fixed, same root cause as the timestamp bug: the login-expiry parse used a bare `DateTime.TryParse` on LAPI's RFC3339 `expire`. Under a mismatched culture that silently fails and falls back to a fabricated `UtcNow + 1h`, pushing re-auth past the real expiry and costing a 401-relogin round trip on every send.
`capacity` now defaults to `1` instead of `0`, matching the one-decision-per-alert shape actually being sent.
## Note on scope
The two 500 causes are independent. On an `en-US` shard only the missing scenario fields were biting; the date bug was latent and would have surfaced as an unexplained regression the first time someone ran a shard under a European locale.
## Verification
The emitted payload is field-for-field identical to a hand-verified request that a live LAPI accepts:
```json
[
{
"scenario": "modernuo/rate-limit",
"scenario_hash": "",
"scenario_version": "1.0",
"message": "ModernUO rate-limit ban for 192.0.2.123",
"events_count": 1,
"start_at": "2026-07-27T15:04:05.123Z",
"stop_at": "2026-07-27T15:04:05.123Z",
"capacity": 1,
"leakspeed": "0s",
"simulated": false,
"events": [],
"remediation": true,
"source": { "scope": "Ip", "value": "192.0.2.123" },
"decisions": [
{
"origin": "modernuo",
"type": "ban",
"scope": "Ip",
"value": "192.0.2.123",
"duration": "300s",
"scenario": "modernuo/rate-limit"
}
]
}
]
```
Regression tests assert the required scenario fields on the **serialized JSON** rather than the DTO — the DTO is not what goes on the wire — and cover the timestamp as a `[Theory]` across `fi-FI`/`th-TH`/`ar-SA`.
`dotnet test --filter "FullyQualifiedName~CrowdSec"` → **21/21 passed**, build clean with 0 warnings.
## Summary
Two related fixes on the outbound path:
1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it.
2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state.
## 1. Send-path stall (RIO)
RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued.
Measured on a 50ms-RTT production shard:
| | before | after |
|---|---|---|
| in-game latency, data flowing | **101–146 ms** | **48–51 ms** |
| p95 | ~135 ms | 52.8 ms |
| samples > 70 ms | 20 | **0** |
The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did.
Two things worth recording, because they explain why this went unnoticed:
- As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones.
- The same measurement at loopback RTT is **microseconds**, so local testing could never surface it.
New settings, both restart-time:
- **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`.
- **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling.
## 2. Send buffer full
`NetState.Send` had three failure modes once the buffer filled, none of them visible:
| writable | behaviour |
|---|---|
| `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect |
| `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way |
| `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** |
The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space.
`Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`.
## Testing
`NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical.
Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8.
## Notes for reviewers
- Upstream change: modernuo/IORingGroup#9.
- The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise.
- Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
### Summary
Moves inventory insurance out of `Mobile`/`PlayerMobile` into its own system at `Projects/UOContent/Engines/Insurance/`, wires it into the feature flag system, and makes disabling it actually disable it everywhere.
### Changes
**New `Server.Engines.Insurance.Insurance` system**
* Owns its own `Configure()`, seeding from the existing `insurance.enable` setting (default `Core.AOS`), so no config migration is needed. `Mobile.InsuranceEnabled` is gone, along with its line in `ExpansionConfiguration`.
* `CanInsure`, `GetInsuranceCost`, `ToggleItemInsurance`, `AutoRenewInventoryInsurance`, `CancelRenewInventoryInsurance` and `OpenItemInsuranceMenu` move here from `PlayerMobile`, which keeps four one-line shims for the context-menu callbacks.
* Every entry point is gated on `Insurance.Enabled`, and the death-time state is only allocated when insurance is on — a shard without insurance pays nothing for it.
**Feature flag integration**
Insurance is now a first-class feature flag: `ServerFeatureFlags.InsuranceEnabled`, registered under the `insurance` key in `FeatureFlagManager.SyncStaticFlag`, so it can be inspected and toggled through the normal flag command/gump rather than only at boot. `Insurance.Enabled` reads through to the flag, so there is one source of truth for every consumer.
**Fixes a memory leak from PvP**
`PlayerMobile.m_InsuranceAward` was a `Mobile` field assigned on every death and never cleared, so every player permanently pinned a strong reference to the last player who killed them. Killers were kept alive by their victims indefinitely.
Death-time insurance state now lives in a `Dictionary<Mobile, InsuranceContext>` owned by the insurance system: the entry is created in `OnBeforeDeath` and removed in `OnDeath`, so nothing outlives the death that created it.
**Removes insurance fields from every PlayerMobile**
`m_InsuranceAward`, `m_InsuranceBonus` and `m_NonAutoreinsuredItems` were carried by every `PlayerMobile` whether or not the shard ran insurance. All three are gone; the equivalent state is allocated per-death, only for players who actually die with insured items, only when insurance is enabled.
**Stale `Insured` flags are inert when insurance is off**
`Item.Insured` is a persisted flag, so items stay marked after a shard turns insurance off. Every read path now checks the flag first, so those items behave exactly as if they were never insured:
* `Item.CheckBlessed` / `Item.IsStandardLoot` — they drop again instead of acting blessed
* `Item.AddLootTypeProperty` — no more phantom "insured" tooltip
* `PlayerMobile.FindItems_Callback` — not yanked out of nested bags on death
* `DestroyEquipment` — no longer immune
* `ClothingBlessDeed` — no longer reports "that item is already blessed"
**Gumps promoted out of `PlayerMobile`**
`ItemInsuranceMenuGump`, `ItemInsuranceMenuConfirmGump` and `CancelRenewInventoryInsuranceGump` were private nested classes reaching into `PlayerMobile` privates. They are now public types in `Engines/Insurance/Gumps/`, talking to the insurance system through its public API. `ItemInsuranceMenuGump.ToggleSelected()` replaces the confirm gump's reach-in to the parent's `_items`/`_insure` arrays.
### Behavior changes
* The per-item "You lack the funds to purchase the insurance" message on failed auto-renewal is no longer sent during death; players get the single 1061115 summary instead. Marked with a TODO pending a decision on whether the per-item message should spam.
* The killer's insurance bonus is deposited once at the end of death processing rather than 300 gold at a time per insured item, and the "gold has been deposited" message is now conditional on the deposit succeeding. Same total.
### Drive-by cleanups
`PoisonImpl.IncreaseLevel` -> `Poison.IncreaseLevel`, a redundant `is NetState { } ns` pattern, `new List<Item>(Items)` -> collection expression, alignment of the `SyncStaticFlag` switch arms, and some comment/formatting fixes in `PlayerMobile`.
## Summary
`AdvancedSearchThreadWorker.Execute` signals `_stopEvent` **before** clearing `_pause` and **before** reading the exit condition. `Sleep()` unblocks the instant that signal fires, so the owning thread can begin the next cycle while the worker is still finishing the previous one — and the worker's two trailing operations then land on the new cycle's state.
`SerializationThreadWorker` already orders the same handshake correctly and documents why (`Projects/Server/Serialization/SerializationThreadWorker.cs`):
```csharp
// The owning thread may start another pause cycle the moment _stopEvent is set
// (Exit does exactly that). Clear _pause and sample the exit condition before
// signaling, or the new cycle's pause request is clobbered / its Sleep orphaned.
var exiting = Core.Closing || worker._exit;
Volatile.Write(ref worker._pause, false);
worker._stopEvent.Set();
```
This applies the same ordering to the search worker. Three lines; no behavior change on the happy path.
## The two failures
**Reuse hang.** The next cycle's `Wake`/`Push`/`Sleep` writes `_pause = true`, then the worker's stale `_pause = false` lands on top of it. The inner loop never observes `pauseRequested`, its queue is already empty, and it spins on `Thread.Yield()` forever — so the owning thread's next `Sleep()` waits on a `_stopEvent` that is never set again. A single search wakes each worker exactly once, so this only surfaces once `_threadWorkers` is reused by a later search.
**Orphaned `Exit()`.** `Exit()` sets `_exit`, `Wake()`s, then `Sleep()`s — the moment the drain's `Sleep()` returns. Reading `_exit` *after* the signal, the worker can observe that fresh `_exit`, return without ever consuming the `Wake`, and leave `Exit()`'s `Sleep()` waiting on a signal nobody will send. The `_thread.IsAlive` guard doesn't close this: the thread passes the check and returns immediately after.
## Verification
Verified with two throwaway timing tests — 25k reuse cycles and 2k drain-then-`Exit` cycles, each under a bounded wait:
| ordering | result |
|---|---|
| previous | `Failed: 2, Passed: 3` — both reproduce, cleanly at the 20s bound |
| this PR | 3 consecutive runs, 5/5, ~0.6s |
**Those tests are deliberately not included.** Their reproduction threshold is a property of one machine's scheduler — at 2k and 200 cycles the buggy build passed — so as permanent tests they'd cost ~560ms and 2000 thread creations on every suite run for a guarantee that may not hold on a CI runner. The ordering is protected the same way `SerializationThreadWorker`'s is: by the comment at the call site.
`UOContent.Tests`: **597/597**.
## Problem
Every map's `.swb` step cache was opened, indexed and logged **twice** on boot.
`MovementPath.Configure()` explicitly called `PathCacheCommands.Configure()` and `CacheEvictionTimer.Configure()`. Both are types exposing a public static parameterless `Configure()`, which `AssemblyHandler.Invoke("Configure")` already discovers and calls once each (`AssemblyHandler.cs:157`). So `PathCacheCommands.Configure()` ran twice, and `AutoLoadAtStartup()` with it. `PathCacheCommands.Configure()` called `PathfindRecorder.Configure()` the same way.
`TryOpenLazyReader` disposes the prior reader before replacing it, so there was no handle leak — but the header and full chunk index of each `.swb` were read twice (~48 MB of files across six facets). The expensive `.mul` hashing was already memoized, so it was not doubled.
## Fix
Consolidate the cache lifecycle into `Initialize`:
- `Configure()` keeps only settings and command registration.
- `Initialize()` opens the readers once, then prebakes only maps that still lack one.
- The post-bake reopen is per-map instead of a blanket `AutoLoadAtStartup()` — on a partial bake (some valid `.swb`, one stale) that would close and reopen the readers already open, a second double-open on a different path.
`Initialize` is the correct phase. `Configure` runs before `TileMatrixLoader.LoadTileMatrix()` and `World.Load()` (`Main.cs:458/460/463/465`), so opening a `.swb` there forced the lazy `Map.Tiles` property — the fingerprint hashes the map files — and built every `TileMatrix` ahead of the loader that owns it, possibly before `TileMatrix.Configure()` settled `Pre6000ClientSupport`. Both sit at the default call priority and the phase sort is unstable. Moving pathfinding out leaves nothing in `Configure` that touches `Map.Tiles`, closing that hazard; the other 22 `.Tiles` users in UOContent are all runtime paths.
Multis stay out of the bake by design — houses and boats are player data that moves, handled by the multi-aware path at query time.
## Logging
The per-map `StepCache: opened ... chunks indexed` line drops to `Debug`. Opening is the expected case; `BakeMap` already logs a rebuild at `Information`, and `Initialize` still emits `PathBake: pre-bake complete (N map(s) written)`.
## Verification
- `dotnet build Projects/UOContent` — 0 errors, 0 warnings.
- `dotnet test --filter FullyQualifiedName~Pathfinding` — **123 passed, 0 failed**.
Boot logs should now show one `opened` line per map at `Debug`, none at `Information`.
Reshapes IP banning around one idea: **core owns the question, content owns every answer.**
Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter.
## The seam
```csharp
public interface IConnectionFilter
{
string Name { get; }
void Configure();
void Start(CancellationToken token);
void Stop();
bool ShouldDeny(IPAddress address);
}
```
The accept path went from hardcoded branches to one question:
```csharp
else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy))
{
logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy);
}
```
Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS.
Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's.
A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either.
This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic.
## What ships behind it
**`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands.
**`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall.
The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up.
The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves.
**`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream.
## CrowdSec: contribute-first
`IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`.
Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work.
### Why not pull decisions from CrowdSec?
The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce.
The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline.
## Threading policy
`CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem:
- Anything touching game state runs **only** on the main loop.
- Heavy work that *needs* game state must be **chunked** across ticks, never threaded.
- Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**.
Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it.
## Shared primitives
`SortedRangeIndex<T> where T : IBinaryInteger<T>` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it.
Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests.
`IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated.
## Config
| File | Owner | Keys |
|---|---|---|
| `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` |
| `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` |
| `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` |
| `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) |
Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change.
## Notes for review
- **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex<UInt128>`, same semantics and public API.
- **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`.
- **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core.
- **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel.
- `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR.
- The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message.
## Tests
**1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader.
## Summary
Hardens the **Advanced Search** engine (`Projects/UOContent/Engines/Advanced Search/`) — the GM entity finder that fans searches across background worker threads. A code review surfaced 14 defects (A–N), including a shard-crasher reachable from a single admin typo and a path that silently disables autosave for the rest of the shard's uptime. Each behavioral fix ships with a test.
Full `UOContent.Tests` suite: **530/530 green** (21 new AdvancedSearch tests).
## Fixes
### Crash / data-loss
- **A — Shard crash on a malformed Property Test.** `AdvancedSearchThreadWorker.Execute` had no `try/catch` and the worker `Thread` is foreground, so a parse throw (`Hits>abc`, `Layer=onehanded` — `Enum.Parse` was case-sensitive, `Hits>1@` — empty sub-expression indexing) terminated the process. Now: `ParseValue`/`CompareValues` use `TryParse`/`Enum.TryParse(ignoreCase)` and return no-match instead of throwing; the per-entity filter is wrapped in `try/catch` (logs + skips); empty expressions are guarded.
- **C — Overlapping searches corrupt state + brick autosave.** `_threadWorkers`/`_threadId` were `static` but `DoSearch` is an instance method; a second search (double-click / two admins) stomped shared worker state and could leave a drain waiting forever on the shared `AutoResetEvent`, so `AutoSave.SavesEnabled` was never restored. Now: an `Interlocked` re-entrancy guard rejects concurrent searches.
- **G — Autosave restore not guaranteed.** The restore lived only in the success callback. Now it's in a `finally` (plus an outer `catch` covering the synchronous setup and a `catch` on the drain body), so autosave + the guard are always released.
### Wrong results
- **D — `@`/`|` operator precedence.** `a@b|c` evaluated as `a && (b || c)` instead of `(a && b) || c`. OR now binds looser than AND (`AdvancedSearchUtilities.EvaluateBoolean`, unit-tested).
- **E — Descending sort, partial last page rendered blank** (the index decreased in descending mode and the `break` early-out killed the loop). Now a bounded `VisibleCount`-driven loop renders the last page in both directions.
- **F — Deleted entities** were not skipped (ghost rows). Now `DoEntitySearch` skips `entity.Deleted`.
- **N — Reference-type comparisons** threw (`Comparer<T>.Default.Compare` on non-`IComparable`) and compared references to a string. Now equality is by value and ordering is guarded to `IComparable` (no throw).
### Worker perf / hardening
- **H** busy-spin → `Thread.Yield()` in the drain; **I** `GetProperties()` cached per `Type`; **J** `HandleValidInternal` moved behind the cheap map/range/region filters; **K** worker threads are `IsBackground` + `Exit()` tolerates an already-terminated worker; **L** `_filter == null` guard; **M** consistent `Volatile` access on `_pause`/`_exit`.
### Documented
- **B** — the residual worker/event-loop read race is documented on `AdvancedSearchThreadWorker`: workers read live entity state concurrently with the loop, so value-type reads may be stale-but-safe and getter exceptions are swallowed; fully eliminating it would require snapshotting entity fields on the main thread (deferred).
## Notes
- New test-only seams (`TryBeginSearch`/`EndSearch`/`IsSearchInProgress`/`VisibleCount`/`TryParseValue`/`EvaluateBoolean`) are `internal` via the existing `InternalsVisibleTo("UOContent.Tests")`.
- Dead `public ParseValue<T>` removed.
- `ConcurrentDictionary` for the reflection cache is intentional — these workers are genuinely parallel.
`ObjectPropertyList.AppendFormatted<T>(value, format)` treated **any** `{value:#}` as the cliloc marker (emitting `#<value>`). But cliloc numbers are integers — a `float`/`double`/`decimal` formatted with `#` is the standard custom-numeric (`#` = digit placeholder) format, not a cliloc reference, so those were being mis-marked.
Gate the marker on an integer value type:
```csharp
if (format == "#" && value is int or uint or long or ulong or short or ushort or byte or sbyte)
```
Now `{someFloat:#}` formats normally (passes `#` through to `TryFormat`); the marker/standard-format ambiguity narrows to the harmless `{0:#}` **integer** case (`#0`). Existing `AddLocalized(int)` / `{value:#}` (all `int`) are unaffected.
Adds `ObjectPropertyListSpanAddTests.HashFormat_OnlyMarksIntegers`: `int {value:#}` → `#<value>`; `double {value:#}` → `42.0.ToString("#")` (`"42"`, no `#`).
Reduces the world-save freeze window from ~740ms to ~78ms (measured on a synthetic 10M-entity / 1.7GB world, 24 cores, through the real pipeline classes) by removing the per-entity handoff between the game loop and the serialization workers, fixing how large indivisible payloads are scheduled, rewriting the BufferWriter hot path, removing per-entity placement state entirely, and finally replacing the global serialized-types tracking with a per-file type table (idx v4) that also shrinks idx files by ~21% and speeds the background write phase. The pipeline has also been validated end-to-end on live-copy worlds in the multi-million-entity range, where the freeze is drain-bound (real `Serialize()` costs far more CPU per byte than synthetic writes) — the same structural wins hold, and entity/file round-trips are byte-clean across both load paths.
## The problem
The freeze window is `max(main-thread handoff, slowest worker drain)`:
1. **The producer was the bottleneck.** The main thread round-robined every entity through per-worker `ConcurrentQueue`s — two interlocked ops per entity, ~740ms of freeze floor at 10M entities before any serialization happened.
2. **Round-robin distributes count, not cost.** "Deep" systems (50MB generic persistence blobs) and "thick" entities (100K-item storage keys) landed on arbitrary workers, producing lopsided drain times on large worlds.
3. **Worst-case scheduling.** `GenericEntityPersistence.Serialize` pushed its self-payload *after* all entities, and generic persistences sort last in the registry — so the biggest indivisible blobs started serializing at the very end, extending the freeze by their entire duration.
## The fix
**Commit 1 — chunked handoff + LPT scheduling + heap pre-sizing:**
- Pooled 4096-entity chunks published to one shared queue; workers pull chunks and load-balance dynamically (a worker busy with a thick entity simply takes fewer chunks).
- `Persistence.SerializeAll` pushes systems largest-first (LPT) using the previous save's payload size (or loaded file size on first boot); self-payloads get dedicated single-entity chunks so they overlap the entity stream instead of ending it.
- Worker heaps pre-size from the loaded save's `.bin` totals, eliminating copy-on-grow inside the first save's freeze.
- `SpinWait` backoff in the drain loop (never `Sleep(1)`), per-worker balance stats logged in debug builds (the call site is compiled out of Release), 1MB snapshot write buffer.
**Commit 2 — workers iterate the dictionaries directly + main thread joins the drain:**
- `GenericEntityPersistence` publishes 4096-slot ranges over its dictionary's backing entries array; workers serialize occupied slots (`value != null`) directly through a `ShadowEntry<TValue>` struct mirroring the runtime's private `Entry` layout. Safe because the dictionary is frozen during `Saving` (mutations divert to the pending safety queues).
- The layout is **proven at startup before any code reads through it**: validation measures the true `Entry` stride via precise allocation accounting (guaranteeing all shadow reads are in-bounds), then verifies every key/value of a churned, resized, freelist-exercised dictionary — reading value slots as raw pointer bits only, never materializing a managed reference until the layout is proven. If a future runtime changes `Dictionary` internals, validation fails with a logged warning and saves fall back to the (fully maintained) enumerate-and-push path.
- The main thread joins the drain via an inline worker after publishing, instead of idling — worth a full worker share, proportionally more on low-core hosts.
**Commit 3 — 2.2x faster BufferWriter write path, single-pass short strings:**
- PGO already devirtualizes and inlines every `IGenericWriter.Write` callsite (interface vs concrete measured identical) — the real per-write cost was the non-inlinable `Index` setter (range-check throw path + per-write high-water tracking) plus span bounds checks. Writes now reserve capacity once, then do an unaligned store through a ref with a raw index increment; the high-water mark folds at Seek/Resize instead of per write.
- Class-level implementations of the hottest default interface methods keep nested writes inlined (a DIM re-dispatches on `this` even at a devirtualized callsite).
- Strings of 85 chars or fewer encode once into a stack scratch instead of walking the string twice (`GetByteCount` + `GetBytes`). Byte output is identical.
- Measured: 34.4 → 15.7 ns/entity on a generated-style write mix; end-to-end freeze ~99ms → ~74-82ms.
**Commit 4 — branch-free fallback push loop:**
- A bare `foreach { PushToCache(entity); }` runs at 2.3ns/entity; the same loop carrying a per-entity heavy-check runs 2.3x slower — the cost is the fatter loop body defeating tight-loop codegen. Entity-level >1MB payloads are rare enough to ride in shared chunks; system self-payloads (the large ones) are still explicitly scheduled largest-first.
**Commit 5 — drop the 9-byte per-entity placement state; snapshots write from worker segment logs:**
- Every `ISerializable` carried `SerializedThread/SerializedPosition/SerializedLength` so `WriteSnapshot` could gather each entity's bytes from the worker heaps in dictionary order. But the idx records absolute positions — bin order is free — so the snapshot is now written in worker-heap order and the join inverts: workers log segments (owner, slot range, heap start) plus one length per record as they serialize; positions are implicit because a worker's writes are contiguous, and identity comes from re-walking the same snapshot slots in the same order (stable until `PostWorldSave`).
- Chunks are persistence-homogeneous (the partial chunk publishes at each `SerializeAll` boundary) so segments route to files by owner with zero per-entity state. Self-payloads keep placement as three private fields on the handful of persistence instances.
- Net: 9 bytes (plus padding) of resident state removed from every item, mobile, guild, and account on every shard; three interface-property stores per entity leave the drain hot path (stamping dirtied one cache line per entity mid-freeze — the lengths log is one sequential stream); each segment's bytes hit the bin as a single span write instead of one copy per entity, speeding the background write phase; and `IGenericSerializable` shrinks to just `Serialize(IGenericWriter)`. Transient cost: ~4 bytes per entity in pooled per-worker logs, released after each write. The save format was unchanged at this point (idx v3, same loader); the v4 bump comes later in the branch.
**Commit 6 — staged file writes replace memory-mapped snapshot writing:**
- `MemoryMapFileWriter` is removed. `FileBufferWriter` composes through the full `BufferWriter` raw write path into a pooled staging block that drains to the file as large sequential positional writes (`RandomAccess.Write`); seeks flush the block and move the file offset, so backwards patches (the idx entity count) become small positional writes.
- Memory-mapped composition paid a soft page fault on every composed page plus unpredictable dirty-section teardown stalls at dispose — measured ~4x slower end-to-end than staged writes at snapshot sizes.
**Commits 7–11 — idx v4: per-file type table replaces SerializedTypes.db and all runtime type tracking:**
- Previously every `Write(Type)` from every worker enqueued into a shared `ConcurrentQueue<Type>` during the freeze (interlocked writes on a shared cache line, millions of mostly-duplicate entries), the background phase drained and deduped it all into a `HashSet`, and the snapshot recomputed `xxHash64(Type.FullName)` once per entity record (~5.4M redundant hashes per save on a large world) to write 9-byte tag+hash idx records plus a global `SerializedTypes.db`.
- The db's only real job was diagnostics: the string name behind "Type `<X>` was not found. Delete all of those types?" during idx loading. That map now lives in the idx itself: each `GenericEntityPersistence<T>` keeps an insertion-ordered `Type -> ushort` table, hydrated at `AddEntity` and on every deserialize path — one dictionary `TryAdd` per entity add on the game thread, amortized across gameplay, and provably immutable while the background writer reads it (adds divert to the pending queues during saves).
- idx v4 layout: the table (names only) is written before the records; records reference it by 2-byte index, shrinking from 33 to 26 bytes (−21%). The loader resolves each table name **once** (`FindTypeByHash(ComputeHash64(name))` — semantically identical to v3 resolution, `TypeAlias` included) into a constructor array, and each record becomes an array index instead of an 8-byte hash read plus dictionary probe. The unresolved-type prompt now surfaces once per type, with the name.
- Deleted outright: `World.SerializedTypes`, the drain/dedupe pass in `WriteFiles`, `BufferWriter`'s type tracking (its `Write(Type)` is now pure — payload format unchanged: tag byte + xxHash64), `FileBufferWriter`'s typeSet parameter, `Persistence.WriteSerializedTypesSnapshot`, and the adhoc db write. SerializedTypes.db is no longer produced.
- **Backward compatibility:** v0–v3 saves (including their SerializedTypes.db and legacy tdb files) load exactly as before, and every legacy load path hydrates the new table so the first v4 save after an upgrade is complete. Stale db files in existing save folders are simply ignored. Verified live: a v3 save boots, saves as v4 (Items.idx −20.2% on a dev world), and reloads with identical entity counts.
## Measured (synthetic 10M entities / 1.7GB, 64+64+32MB system blobs, 24x 2MB thick entities, dense write profile, 24 cores)
| Metric | Before | After |
|---|---|---|
| Steady-state freeze | ~740 ms | **~78 ms** |
| Main-thread publish cost | ~740 ms | **~0.1 ms** |
| Steady-state allocations | 0 | 0 (by iter 2) |
| Worker byte-load spread | 2x | ~1.15x |
The freeze is now bound by pure serialize throughput (payload / cores).
## Tests
- 779 Server.Tests + 501 UOContent.Tests pass.
- New across the branch: chunk fill/flush/owner-boundary tests, pool reuse/clear tests, an end-to-end multi-worker drain through the real wake/push/flush/pause protocol, a 50K-entry churn equivalence test for the shadow iteration re-walk (the exact pairing the snapshot writer relies on), byte-level BufferWriter output/position pins, `RuntimeLayoutIsSupported` so a silent fallback on a future runtime upgrade fails loudly in CI, and a full snapshot **round-trip test** that serializes 25K entities plus a self-payload through real workers, writes the idx/bin from the segment logs, and reloads them through the standard loader (now in v4 format).
- For idx v4 specifically: `FileBufferWriter` staging/drain/seek-patch and oversized-item tests, type-table registration tests, a hand-written v4 fixture proving an unresolvable type name skips only its own records through the console confirmation flow, and a hand-written **legacy v3 fixture** proving old saves still load and hydrate the type table for their next save.
## Trade-offs
- Chunk scheduling is nondeterministic, so worker heaps ratchet to each worker's max-ever draw rather than a fixed share. With slot ranges the balance is tight (~1.15x), so the effect is small; a shared slab pool remains an option if production shows retention creep.
- Entity-level heavy items inside slot ranges are serialized wherever they're encountered (no LPT for them); worst-case tail is one thick entity's serialize time (~ms). System self-payloads — the large ones — are still explicitly scheduled largest-first.
- Snapshot-write error granularity is per segment rather than per entity (heap-bounds bugs were the only thing the per-entity catch ever caught; idx metadata reads keep per-record granularity).
- idx v4 is a save-format version bump: old saves load unchanged through the preserved legacy paths, but saves written by this branch require this loader. Per-persistence type tables cap at 65,535 distinct entity types per boot (hard throw, orders of magnitude of headroom), and a type's table slot persists until restart even if its last entity is deleted — a few stale name entries per file, by design.
Two stalls hit CI in one night with nothing bounding them but GitHub's 360-minute default:
1. A silently deadlocked test host (2h41m before manual cancel) — the code fix is on #2525; this PR adds the guardrails that make any recurrence cheap and self-diagnosing.
2. A dnf step stalled mid-download for 12+ then 30 minutes. Probing the EPEL mirror pool directly: the **first** mirror in the current US mirrorlist returns consistent HTTP 500, and sibling mirrors serve mismatched metadata generations, while Fedora's status page reads all-green (it only tracks central services, not the volunteer pool).
## Changes
- **`timeout-minutes: 30` on both jobs.** Verified live: the cap killed a stalled job at exactly 30:02 instead of 6 hours.
- **`dotnet test --blame-hang-timeout 10m --blame-hang-dump-type full`** — a stuck test host is killed after 10 minutes and vstest names the in-flight tests with a full process dump; `TestResults` (including dumps) upload as artifacts on failure. A future hang produces a stack trace instead of a bill.
- **EPEL setup follows the quickstart ordering**: `dnf-plugins-core` → enable CRB → `epel-release`. `epel-next` is no longer installed — none of the prerequisites need it (validated green on Stream 9), EPEL 10 doesn't have it, and it's one more mirrorlist to fetch.
- **Matrix**: adds **Ubuntu 26.04 LTS**, **CentOS Stream 10**, and **AlmaLinux 10** (real-EL10 coverage); bumps **Fedora 42 → 44** (42 went EOL in May). README badges and the Linux prerequisites section updated to match.
Kept deliberately simple per review: no retry wrappers around dnf — mirror hiccups are rare and the job cap bounds the damage.
## Problem
On headless Linux deployments (systemd service, Docker without a TTY, `nohup`), the ModernUO process pegs a full CPU core even when idle. It does not reproduce on Windows because that runs with an interactive console.
## Root cause
`ConsoleInputHandler` runs a background thread (named "Console Input Handler") that loops on `Console.ReadLine()`. When stdin is **not** an interactive terminal, `Console.ReadLine()` returns `null` at end-of-stream **immediately** on every call, so the loop `continue`s in a tight spin — one core at 100%.
Reproduced in a container running the actual distribution: the "Console Input Handler" thread sat at ~90% CPU on a headless boot; with a blocking stdin it dropped to idle.
## Fix
1. **Detect headless once at startup:** `Core.Headless = Console.IsInputRedirected`.
2. **Extract a testable `ConsoleInputPump`** that owns the input stream: per line read, it *atomically* (under one lock) either delivers the line to a waiting prompt or dispatches a console command, and it **ends on EOF instead of spinning**. Cleanup runs unconditionally in a `finally`, so a pending prompt is always released (never hangs). Replaces the old `async void` loop and the fragile `_expectUserInput` / two-`AutoResetEvent` / `_input` handshake.
3. **`ConsoleInputHandler` becomes a thin headless-aware facade** over the pump. Headless: the reader thread never starts (`Console input disabled (headless: stdin is not a TTY).`), and `ReadLine()` throws a fatal `HeadlessConsoleInputException`.
4. **Data-gating and first-boot prompts** (deserialization "delete bad types? y/n", save-conflict, config/expansion setup) now route through `ConsoleInputHandler.ReadLine()`, so a headless server crashes fatal with a clear message instead of reading `null` (previously an NRE or a silent wrong branch).
Design decision (model b): headless servers are expected to be supplied with configuration/save data (including the owner account); interactive prompts when headless are fatal by design.
## Testing
- New `ConsoleInputPumpTests` (5 tests): EOF ends the loop without spinning; command dispatch; a pending prompt receives the next line; EOF while a prompt is pending completes it with `null` (no hang); a throwing command lookup does not hang a pending prompt. The tests synchronize on real pump state (no `Thread.Sleep`), so they are deterministic on slow CI.
- Full `Server.Tests`: no new failures introduced.
## End-to-end verification (Docker, real distribution)
| | Console Input Handler thread | Container CPU |
|---|---|---|
| Before fix (headless boot) | ~90% | ~199% (2 cores) |
| After fix (headless boot) | **not started** | **~11%** |
After the fix, a headless boot logs `Console input disabled (headless: stdin is not a TTY).`, loads the world normally, and idles instead of spinning.
## Problem
Items dropped on the ground never decay. Corpses do, which makes the breakage look selective — but corpses are unaffected only because `Corpse.BeginDecay` runs its own `InternalTimer` and never touches `DecayScheduler`. Ordinary items are the only things that depend on the scheduler.
## Root cause
`Item.MoveToWorld` called `SetLastMoved()` — which triggers `UpdateDecayRegistration()` — at the *top* of the method, before detaching the item from its parent and before assigning the new map. `CanDecay()` reads `Decays`, `Parent`, **and** `Map`, so registration was evaluated against the item's *pre-move* state.
Because the `Item` constructor sets `m_Map = Map.Internal`, and `Mobile.Lift` calls `item.Internalize()` to put an item on the cursor, registration was consistently one step behind:
| State | Tracked for decay? | |
|---|---|---|
| Item on the ground | **No** | never decays |
| Item held on the cursor | **Yes** | backwards |
Nothing corrected it afterwards: the later `m_Map = map` assigns the field directly, bypassing the `Map` property setter, and that setter does not refresh registration either. With no parent, `RemoveItem` (which *does* re-register) never runs.
World load masked this — `ItemPersistence.PostDeserialize` re-registers every item against its final state, so decay appears to work for items that survive a restart. Only freshly dropped items are affected.
**Fix:** stamp `LastMoved` up front so decay math stays correct, then call `UpdateDecayRegistration()` once the parent, map, and location are final.
## Audit of the rest of the call sites
All 16 `SetLastMoved()` call sites were reviewed. `SetLastMoved()` must keep refreshing registration — `LastMoved` feeds `ScheduledDecayTime` and therefore which bucket an item belongs in — but it may only run once parent/map are final. The vendor, house, lift and drop sites already satisfy that. The rest of this PR fixes the ones that did not, plus what the audit turned up:
- **`Item.Deserialize`** stamped via `SetLastMoved()` before the version data was read, registering against an unread `Map`/`Parent`. Safe only by accident (the `Item(Serial)` ctor leaves flags at 0, so `Decays` is false), and it cost an unregister per item per world load. Now stamps only; `PostDeserialize` does the registration.
- **`Item` constructor** registered then immediately unregistered every item — the `Movable` setter saw `m_Map` still null, so `CanDecay()` was true. Also removes a `Configure`-order landmine: constructing an `Item` before `DecayScheduler.Configure()` would have thrown in `Shared.Start()`.
- **`Container.Destroy`** stamped `LastMoved` immediately before `MoveToWorld`, which now stamps it itself.
- **`Unregister` was documented O(1)** but scanned twelve buckets and did a linear `PriorityQueue.Remove`, on every construction, deserialize and move. Items now record where they are tracked in `Item.DecaySlot` (1 byte), so untracked items — the common case — leave in O(1).
- **A refused decay silently dropped the item.** `ProcessActiveQueue` deleted on `OnDecay() == true` but did nothing when a region refused, leaving the item dequeued, untracked and on the ground forever. It now restarts the decay clock; re-registering as-is would spin, since `ScheduledDecayTime` is already past.
## Two content bugs of the same class
The decay system replaced a polling sweep. Under polling, a `Decays`/`DecayTime` override could read live state every pass. Under a registration model it cannot — the scheduler drops items that stop being eligible, but nothing enrols one that becomes eligible while untracked.
- **`TreasureChestLevel1-4`** overrode `DecayTime` as `Utility.Random(15, 60)` — a fresh roll on every read. `ScheduledDecayTime` is read repeatedly (to bucket, to re-bucket on rotation, to test whether due), so those reads disagreed: the chest re-bucketed every tick and decayed early instead of after its intended interval. Now rolled once per chest. Distribution unchanged — `Utility.Random(from, count)` is RunUO's `from + Next(count)`, so this is 15–74 minutes, as before.
- **`StrongBox`** overrode `Decays` with a live check on `_house`, `_owner.Deleted` and `IsCoOwner`. Nothing notifies the box when any of those change, so it was never enrolled and the override never decayed anything — and it could not have: `HouseRegion.OnDecay` refuses a secured item inside a standing house, and the box is in `Secures`. Decay was never the mechanism here.
- A strongbox is only ever its owner's. Without a house, or without an owner still co-owning that house, it would be a free container anyone could loot, so `Validate()` destroys it. The old check missed exactly those two cases — it required a non-null owner and treated a null house as valid. A deleted owner deserializes back as null, which `IsCoOwner` rejects. The now meaningless `Decays`/`DecayTime` overrides and an unhelpful `Console.WriteLine` are gone.
`DecayScheduler` now documents both constraints.
## Tests
`DecayRegistrationTests` (Server) covers world placement, lift/drop, container round-trip, cursor-held (must *not* track), `Container.Destroy` spill, `DecaySlot`/structure agreement, refused decay, and a full decay lifecycle driven through the scheduler. `TreasureChestDecayTests` (UOContent) locks `DecayTime`/`ScheduledDecayTime` stability across all four chest levels.
To make the lifecycle testable deterministically, `DecayScheduler` gains `internal` members (visible only via existing `InternalsVisibleTo`): `IsRegistered()`, `ProcessTick(now)` — extracted from `OnTick()` with no behaviour change — and `ResetForTests()`.
Red/green verified. Without the `MoveToWorld` fix, 5 of 6 of the original tests fail, including `ItemOnGround_ActuallyDecaysAfterDecayTime`, which shows a ground item never decays even after a full simulated hour. Without the chest fix, 8 of 8 chest tests fail. Without the refused-decay fix, that test fails.
Server.Tests 737/737 and UOContent.Tests 509/509 pass.
## The bug
#2522 rewrote the outgoing huffman table in `NetworkCompression.cs` and transposed symbol `0x19`'s code from `0x1CE` to `0x12E` (both 9 bits, so the length distribution — and the Kraft sum — stayed valid, which is why nothing obvious tripped).
The real damage is that it broke prefix-freeness. `0x12E` is `100101110`, and symbol `0x0D`'s 8-bit code is `10010111` — a proper prefix of it. The client's decoder walks the tree bit by bit, so it hit a valid leaf at `0x0D` after 8 bits, emitted the wrong byte, and then reframed every subsequent code.
That is exactly what the reporter's capture shows. Server sends `BF 00 0C 00 19 02 00 00 00 01 00 00`; the client's post-decompression stream reads `BF 00 0C 00 0D 55 00 00 01 00 00` — the literal `0D` is the mis-decoded `0x19`, and the packet is now one byte short, so framing desyncs from there on.
## Impact
Any outgoing packet with byte `0x19` anywhere in its body (serials, coordinates, hues, lengths, text) corrupted the stream. Because the desync is in framing rather than a single field, the client silently stops applying server updates while still being able to send — no disconnect, no error.
`StatLockInfo` (`0xBF` subcommand `0x19`) is sent during login, so it reproduces on essentially every connection. This is also #2526: "can only walk a few steps, then the client stops responding" is the same desync, not a VPS sizing problem.
## Fix
One entry, restored to the canonical value:
```diff
- 0x9, 0x191, 0x9, 0x12E, 0x7, 0x03F, ...
+ 0x9, 0x191, 0x9, 0x1CE, 0x7, 0x03F, ...
```
## Validation of the whole table
Rather than eyeball 257 entries, I diffed the current table against **every revision of it in this repo's history** — all 34, back through the renames to the original import. All 34 agree with each other, and `0x19` is the sole disagreement with #2522's rewrite. No other entry has ever changed.
I also validated the table structurally: all 257 lengths in `[2,11]`, every value fits its declared bit-length, Kraft–McMillan sum exactly 1, and no code is a prefix of any other. It passes on all counts now, and the prefix check is what located the bug in the first place.
Both checks were one-off validation scripts, not committed — see below.
## Test
A single known-answer test (`~10ms`) that compresses all 256 symbols and asserts the exact output bytes. The expected bytes were generated from the canonical table, *not* from the implementation, so the test isn't circular. Any single wrong table entry changes the output, so it pins all 256 entries plus the terminal code, and it exercises the encoder end to end.
A round-trip test would **not** catch this class of bug — encoder and decoder built from the same table agree with each other even when the table is wrong. The contract being violated is with the client's hard-coded tree, so the expected bytes have to come from outside the implementation.
The structural prefix-free check and a second `StatLockInfo` vector were deliberately dropped after they'd served their purpose: the table is now verified and effectively frozen, so the structural check was guarding a constant, and the `StatLockInfo` vector is a strict subset of the all-symbols one. What remains covers the risk that's still live — `Compress` is a hand-unrolled bit-packing loop that will get optimized again, and this is the guard against that rewrite silently corrupting the wire format, which is precisely what happened here.
Verified the test fails when the bug is reintroduced and passes when fixed. Full `Server.Tests` suite green: 727 passed.
Started as an allocation pass over `StepCache` and grew into a cleanup of the surrounding pathing engine. Four commits, each independently reviewable; net **−560 lines**.
Build clean (0 warnings). All 122 `Server.Tests.Pathfinding` tests pass.
---
## 1. `perf`: pool the strata buffer, cut a hot-path dictionary lookup
**The headline is that `TryGetMask` — the actual hot path — was already allocation-free.** `StepMask` is a readonly struct, `StaticTileEnumerable` is a `ref struct`, `ChunkMissState` is a struct in a `Dictionary`. So most of this is a bake-throughput and GC-churn win, with one exception noted below.
`BuildChunk` accumulated packed multi-Z strata into a `List<byte>` that grew by doubling (256 → 512 → 1024 → …) and then paid a final `ToArray()`. A full map bake runs it ~114k times. It now writes into a `byte[]` rented from `STArrayPool<byte>.Shared` through a span writer, and hands the chunk one exact-size copy.
**This required fixing a latent out-of-bounds guard.** The record-fit check reserved headroom for **8** strata (`StratumByteLength * 8`) while `ComputeStandableSurfaceZs` can return up to **16** — so a cell could write 305 bytes starting from a 65,383-byte offset. Against a `List` that was benign (it just grew past 64 KB, and emitted offsets stayed under the `NoStrata` sentinel). Against a fixed-size rented buffer it is an out-of-bounds write, so tightening it was a *prerequisite* for the pooling, not a drive-by. The guard is now exact, which additionally proves no emitted offset can collide with `NoStrata == ushort.MaxValue`.
**One genuine query-path win:** `ShouldPromoteAfterMiss` did *two* dictionary lookups per miss — a `TryGetValue`, then an indexer assignment that re-hashes and re-probes. It now mutates in place via `CollectionsMarshal.GetValueRefOrNullRef`. This runs on every uncached chunk touch during A* expansion. The window-expiry branch keeps its explicit early return, so `MissPromotionThreshold == 1` still resets rather than promoting.
Also dropped `StepProbe.ComputeStrataAt` / `ComputedStratum` (dead code, zero callers) and collapsed six 18-argument `new StepMask(0, 0, …, kind)` blocks into `Fallthrough(kind)`.
**Considered and rejected:** pooling the `Direction[]` that `Find` returns. It *escapes* the call — `MovementPath` holds it across ticks while `PathFollower` walks `m_Index` through it — so it cannot be rented-and-returned, and it cannot be borrowed from the shared `BitmapAStarAlgorithm.Instance` without one creature clobbering another's in-flight path. `CheckPath` rate-limits repaths to one per 2s per creature, putting this at roughly 60 KB/sec at 1,000 pathing creatures. Not worth a public API break plus a use-after-return footgun.
## 2. `docs`: rewrite the comments for publication
The comments had accumulated as development notes: internal phase jargon (`Tier 4`, `the Phase-2 synthesizer`), change narration aimed at a reviewer (`which the old ComputeStandingZ anchor missed`, `legacy behavior`), benchmark anecdotes (`benchmarked as near-optimal`, `a ~20 ns lookup`), and paragraphs restating the code.
Rewritten to keep the rationale you cannot recover by reading the code — why the source-Z guard cannot be widened, why multis fall through with a halo, why the promotion gate counts Finds rather than calls, why `ComputeFingerprint` must hash the *files* and not the live tile tables — and drop the history that got us there.
Three comments were **factually wrong**, not just wordy:
- `CacheEvictionTimer` and `CacheStats` documented a class called `StaticWalkabilityCache`. No such class exists — it is `StepCache`.
- `StepCacheFile` declared `File layout v8` while `FormatVersion` is 9, and called the current record layout "the v6 layout" in four places. The layout descriptions are now unversioned so they cannot drift again.
- `StepProbe.ComputeStandingZ` claimed `StepCache` uses it to bake `SourceZ`. It has not since the baker moved to the clearance-aware `ComputeStandableSurfaceZs`; only a parity test calls it.
## 3. `refactor`: simplify `StepCacheFile.Write`, consolidate the format tests
`SaveToFile` walked `_keysList` **twice** — once to count the map's chunks, then again through a `ChunkEnumerator` closure to emit them — because `Write` needed the count up front to size its index array. Both loops had the same root cause. Passing a **span** collapses them: the count is just `span.Length`.
That deletes the `ChunkEnumerator` delegate, the closure over the list enumerator, and **both `InvalidOperationException` throws**, which existed only to police the delegate's "yield exactly `chunkCount` chunks" contract — a contract a span makes unrepresentable.
`Write` now patches the header's `IndexOffset` by seeking back to it rather than reaching into the writer's live buffer with `BinaryPrimitives`. That also retires `IndexOffsetFieldPosition`, a hand-maintained byte offset that had to track the header layout, and sidesteps the stale-array hazard that motivated the manual patch (`BufferWriter` reallocates on growth).
**Tests:** `StepCacheFileV6/V7/V8Tests` were named for the format version that introduced each transform — and the format is now **v9**, so all three names described formats the loader rejects outright. Beyond triplicated builders and plumbing, two things were actually broken:
- The three near-identical rejection tests each cited a `MinSupportedVersion` that had since moved (`"version 5 < MinSupportedVersion 6"`, `"6 < 7"`, `"7 < 8"`). They passed for the wrong reason.
- `AssertBaseEqual` (used by V7 and V8) **silently skipped the swim and strata trailers**. A regression dropping either would not have failed those tests.
Now one `StepCacheFileFormatTests`, named for behavior — predictive-Z elision, compression, compact index — with a single `AssertIdentical` that does check both trailers, the three rejection tests folded into one theory that also covers a future version, and a zero-chunk case the delegate-based writer never had coverage for.
## 4. `test`: consolidate the parity and lifecycle tests
Three files tested "parity" and none of the names said *which*. They were three different layers, and the seams are the useful part, so they are now one `StepCacheParityTests` that names them:
| Test | Compares | Answers |
|---|---|---|
| `ProbeMatchesSlowPath` | StepProbe vs MovementImpl | Is the bake right? |
| `CacheMatchesProbe` | StepCache vs StepProbe | Is it stored and returned intact? |
| `CacheServesReachableWalkStates` | StepCache vs MovementImpl | End to end, over the states A* visits |
Merging removed a duplicated stub `Mobile`, duplicated region seeds, and a filename/class mismatch (`StepProbeParityTests.cs` declared `StaticWalkabilityParityTests`). `SwimBake_ProducesWetCells` moved with it — it lived in the cache parity file but never touched the cache.
Tests reached into `StepCache._chunks` via `GetField` in **9 places**, each rebuilding the key encoding and cell-index arithmetic by hand. `StepCache` now exposes `GetResidentChunk` and `ResidentIndexInSync` alongside the internal test hooks it already had (`LazyReaderHasChunk`, `CurrentFindGeneration`), and the shared arithmetic moved to `PathingTestSupport`. All 9 reflection blocks are gone.
`StepCacheLifecycleTests` is regrouped by what it covers — promotion gate, fallthrough routes, strata, swim layer, eviction — with the `Tier4*` names dropped. Removed `Singleton_IsAvailable`, which asserted an inline-initialized static property was not null; that is the entire 123 → 122 test-count delta.
---
## Verification
Tests were mutation-checked rather than just run, since round-trip and parity tests can pass while a transform silently no-ops:
- Injecting an off-by-one into the `IndexOffset` patch fails **15 of 123** — the format tests are load-bearing.
- Offsetting the cache's cell index by one fails **7 of 10** parity cases, and the 3 that stay green are exactly the ones that do not touch the cache. The layering localizes a fault rather than just reporting one.
## Summary
`[AddonGen` currently produces addon scripts that **do not compile**, plus a few
gather-logic and UI bugs. This fixes all of them.
## Compile-breaking (verified)
Every generated addon failed to build because of the item-component emission path:
- **Trailing comma + missing semicolon.** Items were emitted as a multi-line
`AddComponent(\n … ,\n)` — a trailing comma in the argument list and no terminating
`;`, i.e. `CS1525: Invalid expression term ')'` and `CS1002: ; expected`.
- **`Deed` missing `new`.** The template emitted
`public override BaseAddonDeed Deed => {name}AddonDeed();` — invoking the type as a
method (`CS1955: Non-invocable member … cannot be used like a method`).
Both are now fixed; components are emitted on a single line matching the existing
static-tile path:
```csharp
AddComponent(new AddonComponent(3215) { Light = LightType.Circle300, Hue = 5 }, 2, 3, 5);
```
**Verification:** compiled the generator's *output* (a representative two-component addon —
one plain, one hued + light-source) before and after the change against minimal
`BaseAddon`/`AddonComponent` stubs:
- Before: `CS1525` + `CS1002` (item path), and `CS1955` in isolation for the `Deed` line.
- After: **Build succeeded.**
`Projects/UOContent` also builds clean with the source change.
## Gather-logic + UI (reasoned from the code, not runtime-tested)
- **Inverted Z-range guards.** The tile/item scan used `if (range && …)`, so with the range
filter off (the default) map tiles and items were never captured — inconsistent with the
Static pass's `if (!range || …)`. Corrected to match.
- **"Export Items" was dead unless "Export Statics" was also checked** — the items scan was
nested inside `if (statics)`. Items now scan independently. Placed `Static` items are
skipped in this path because they're already captured (with hue/light) by the
`GetItemsInBounds<Static>` pass, which also removes a pre-existing double-count.
- **Swapped gump Min/Max labels** — the "Max" label sat over the Min entry and vice versa.
## Notes
The three gather/UI fixes are reasoned from the code rather than exercised through the
in-game gump, so they're worth a close look in review. The compile fixes are the headline
and are output-verified.
Adds the four remaining named Stygian Abyss throwing artifacts as item definitions, completing the throwing artifact set (7/7 now defined in ModernUO).
## Added (stat-for-stat from ServUO)
- **Abyss Reaver** (Cyclone) — random Throwing +5..10 skill bonus, +25..35 damage, Exorcism slayer
- **Storm Caller** (Boomerang) — Hit Lightning / Hit Lower Defense, 20/20/20/20/20 elemental split
- **Banshee's Call** (Cyclone) — Hit Harm / Hit Life Leech, 100% cold, Velocity 35
- **Wind of Corruption** (Cyclone) — Hit Stamina Leech / Hit Lower Defense, 100% chaos, Fey slayer
## Acquisition — deferred (documented)
These are **`[add`-only for now**. Their OSI sources aren't in ModernUO yet:
- Abyss Reaver → the Into the Void quest (Agralem), deferred with the Abyss void-creature content.
- Storm Caller / Banshee's Call / Wind of Corruption → renowned/boss creatures (`WyvernRenowned`, `PrimevalLich`, etc.) that need a `BaseRenowned` framework port.
Per direction, the items are worth defining now; wiring their drops follows later.
## Notes
- Storm Caller's ServUO `WeaponAttributes.BattleLust = 1` is left as a `//TODO Implement BattleLust` — the attribute doesn't exist in ModernUO's `AosWeaponAttribute` set yet.
- The human Bow variant `WindOfCorruptionHuman` is archery, not throwing — intentionally out of scope.
Server builds clean.
Follow-up to the gargoyle Throwing work (#2510/#2512/#2514). Ports two Stygian Abyss creatures from ServUO so two of the throwing artifacts finally have a real OSI drop source instead of being `[add`-only.
## Changes
- **`Raptor`** and **`StoneSlith`** — ported stat-for-stat from ServUO. They **spawn automatically**: `Distribution/Data/Spawns/post-uoml/termur/TerMur.json` already contained Raptor/StoneSlith spawner entries referencing these class names, so no spawn-file edits were needed.
- **`RaptorClaw`** (Boomerang-based) and **`StoneSlithClaw`** (Cyclone-based) artifacts, dropped from each creature's `OnDeath` at ServUO's 0.5% rate (uncontrolled only).
- StoneSlith retains its `GraspingClaw` monster ability; both use `BleedAttack`.
## ModernUO idioms
Default range perception (16) / fight range, derived `GetSpeeds` (no `SetSpeed`), `[SerializationGenerator(0)]` codegen, collection-expression arrays.
## Documented omissions vs ServUO
Flagged as TODO in-code — all are content missing from ModernUO, not silent drops: Raptor's friend-spawn timer + its 25% `AncientPotteryFragments` drop; StoneSlith's `TailSwipe`, `DragonBlood`, and `SlithEye`/`TatteredAncientScroll`/`AncientPotteryFragments` drops; plus `HideType.Horned/Spined` and `PackInstinct.Ostard` (no ModernUO equivalents yet).
## Not included
The other three throwing artifacts (Storm Caller, Banshee's Call, Wind of Corruption) drop from renowned/boss creatures via a `BaseRenowned` artifact-list system that doesn't exist in ModernUO yet — a separate, larger effort. AbyssReaver stays with the (deferred) Into-the-Void quest.
Verified: server builds clean; throwing suite 14/14.
The Throwing skill shipped with `StrScale`/`DexScale`/`StatTotal`/`StrGain`/`DexGain` all `0` in `skills.json`, so training it never raised Str or Dex — unlike every other weapon skill.
Fills those in by mirroring **Archery** (the Dex-primary ranged analog, which matches Throwing's existing `PrimaryStat: Dex` / `SecondaryStat: Str`): `StrScale 0.025` / `DexScale 0.075`, `StatTotal 10`, `StrGain 0.25` / `DexGain 0.75`.
Data-only change.
## Problem
Three coupled issues, each hiding the next:
1. **CI passed despite failing tests, with no test logs.** ([example run](https://github.com/modernuo/ModernUO/actions/runs/28639143286/job/84931544255) — the `Test` step produced zero output and the job went green.)
2. **Two `EmitsLowerStatReqWhenPassed` tests** fail with `KeyNotFoundException: '1060435'`.
3. Once CI actually ran the tests, **~337 UOContent tests failed** with `FileNotFoundException: tiledata.mul was not found` — the test bootstrap force-loaded copyrighted client data that CI doesn't have.
## Root causes & fixes
### 1. CI ran zero tests (`fix(ci)`)
The `Test` step ran `dotnet test --no-restore`, but the `Build` step only restores/builds `Application` — never the test projects. Without a restore, the test projects have no `project.assets.json`, so `Microsoft.NET.Test.Sdk`'s targets aren't imported, they aren't recognized as test projects, and `dotnet test` runs the `VSTest` target against **zero** projects → no output, **exit 0**.
- Both jobs now run `dotnet test --logger trx --results-directory ./TestResults` (test projects restore and run) **plus a guard** that fails the job if no `.trx` is produced — a permanent backstop against silent zero-test passes.
### 2. Impossible OPL tests (`fix(ci)` + `test(opl)`)
#2501 deliberately emits `LowerStatReq` (`1060435`) **inline in each item**, not in `GetProperties`. A follow-up "fix" dropped the `lowerStatReq:` argument to make the tests compile but left the assertions expecting `1060435`.
- Removed the two impossible tests, then removed the **entire `Tests/PropertyList/` OPL attribute set** from #2501: these assert exact cliloc/value/order of OPL emission per item base — a one-time proof of the #2501 rewire, now a permanent tax on modding (any admin reorder/value change/added line reddens the build). The one non-trivial case (LowerStatReq) is what just broke, because the test was wrong. Inline emission stays covered by the `BaseArmor`/`BaseClothing` tests.
### 3. Tile-data-dependent tests crashed CI (`test(uocontent)`)
UOContent.Tests' collection-fixture constructor force-loaded `tiledata.mul` unconditionally. On CI (no client files) it threw, and xUnit failed **every test in the collection** with the same error — mostly collateral (packet/scheduler/spawner tests that don't need tile data).
- Mirror Server.Tests' graceful pattern: `TestServerInitializer` probes for `tiledata.mul` and only loads tile/multi data (and runs the tile-dependent configure steps) when present, exposing `TileDataLoaded` so the fixture no longer throws.
- Add a shared `TileDataRequirement.SkipIfMissing()` guard and apply it to exactly the **31** pathfinding/multi/AI tests that genuinely need real tile data (`[SkippableFact]`/`[SkippableTheory]`).
## Verification (all local)
| Scenario | Server.Tests | UOContent.Tests |
|---|---|---|
| **Client data absent (CI)** | 726 pass, 17 skip, **0 fail** | 469 pass, 32 skip, **0 fail** |
| **Client data present (dev)** | 726 pass, 0 skip, **0 fail** | 501 pass, 0 skip, **0 fail** |
- Full `dotnet test` exits **0**; TRX files produced; the no-test guard trips (exit 1) only when zero `.trx` are produced.
Phase 2 follow-up to #2510 — wires acquisition for the gargoyle Throwing content that Phase 1 deliberately left out.
## Changes
- **SA loot flavor (`IsStygian`)** — completes the dead loot path from the original PR: adds `SAWeaponTypes`/`SARangedWeaponTypes` pools + `isStygian` branches to `Loot.RandomWeapon`/`RandomRangedWeapon`, **and** the missing piece — computes `IsStygian` (`map == Map.TerMur`) in `LootPackEntry.Construct` and threads it through `LootPackItem.Construct`. Ter Mur creatures now roll SA gear + the three throwing weapons (Boomerang/Cyclone/SoulGlaive) from their normal `BaseWeapon`/`BaseRanged` loot entries. Conservative `TerMur`-only (not ServUO's extra `|| RandomBool()`).
- **Valkyrie's Glaive** — re-added as a self-contained Ter Mur stealable artifact at `(843, 665, 27)`, matching ServUO/OSI, with the previously-missing `ArtifactRarity => 5`.
## Deferred (documented)
The 5 host-dependent artifacts (Raptor Claw, Stone Slith Claw, Storm Caller, Banshee's Call, Wind of Corruption) are intentionally NOT included — their OSI drop sources (Raptor, StoneSlith, and the renowned/boss creatures + a `BaseRenowned` artifact-list system) don't exist in ModernUO yet, so re-adding the items now would leave them `[add`-only. They'll follow a dedicated SA-creatures effort.
## Notes
- No new tests: both changes are property/plumbing with RNG-driven output — no deterministic complex logic to assert (consistent with the repo's test-scope conventions).
- Verified: server + test project compile; throwing suite 14/14.
Supersedes #2376 (@jwvalentine). This is a reviewed, corrected, and scoped-down **Phase 1** of Joe Valentine's throwing implementation — his original commits are cherry-picked here with authorship preserved, plus a fix/scoping pass. Phase 1 lands only the **core gargoyle Throwing skill**; the incomplete content is excised for follow-up PRs (see below).
## What's included (core skill)
- `BaseThrown` combat mechanics on top of the existing skeleton: close-quarters penalty, below-min-range penalty, shield penalty, STR-scaled range, overthrow damage penalty.
- Base weapons: **Boomerang, Cyclone, SoulGlaive** (gargoyle-only), + blacksmith crafting (SA-gated).
- Two symmetric hooks on `BaseWeapon` (`ModifyHitChance`, new `ModifyDamage`) that are inert no-ops for every other weapon.
## Fixes over the original
- **Overthrow damage**: was dead code (the swing gate already guarantees you're within `MaxRange`, so the old `ComputeDamage` check never fired). Reimplemented as `finalDamage × 0.53` applied *after* all offensive bonuses via a new `ModifyDamage` hook, firing at the outer range ring.
- **`DefMaxRange`**: clamped to `[MinThrowRange, MaxThrowRange]` (uncapped before → e.g. range 13 at 200 Str) and guarded against a latent divide-by-zero.
- **Close-quarters mitigation** now uses `RawDex` (matches ServUO/OSI; deterministic under stat mods).
- **Return-throw timer** guarded against a deleted/unmapped thrower/target.
- Reverted the `MovingShot` change (it rebalanced archery — belongs in a separate PR).
- Kept only complex-logic tests (hit-chance/range/damage math); dropped property-value assertions.
## Excised for follow-up PRs (Phase 2/3)
7 named artifacts, the Into-the-Void quest + Agralem, GargishOutcast, the Bladeweaver vendor, and the SA loot tables — these were unwired/non-functional (loot never triggered, quest/creature never spawned) and will return properly wired. `StormCaller` also needs its missing Battle Lust, and the quest its correct void-creature target.
## Verification
- Build: 0 warnings / 0 errors.
- `UOContent.Tests`: **501/501** passing (the `ModifyDamage` hook causes zero regressions across all weapons).
- Full whole-branch review completed: no must-fix defects.
## At a glance
```csharp
public override void GetProperties(IPropertyList list)
{
base.GetProperties(list);
// Accumulate any number of free-text lines. On dispose the block flushes via
// AddChunked, splitting across as many OPL properties as needed so none can
// overflow the legacy 2D client's per-property buffer (which would crash it).
using var block = list.TextBlock();
if (luck > 0)
{
block.Add($"Luck Bonus: +{luck}%"); // zero-alloc interpolation
}
block.Add("Cannot be repaired".AsSpan()); // plain text, no string allocation
// Already holding a '\n'-joined string? Skip the builder and chunk directly:
// list.AddChunked(description);
}
```
## What
Adds a safe path for emitting **variable-length, free-form (non-cliloc) tooltip text**:
- **`ObjectPropertyList.Add(ReadOnlySpan<char>)`** overloads — append raw text with no string allocation. Makes the old single-arg `Add(string)` redundant (a `string` binds to the span overload implicitly), so it's dropped.
- **`AddChunked(ReadOnlySpan<char>)`** on `IPropertyList` — splits newline-joined text at `\n` boundaries across as many passthrough-cliloc properties as needed, so no single property exceeds the cap.
- **`OplTextBlock`** (`ref struct`) + **`IPropertyList.TextBlock()`** — an ergonomic builder that accumulates `\n`-joined lines (with a zero-alloc interpolated `Add($"...")` overload) and flushes via `AddChunked` on dispose. Usage: `using var block = list.TextBlock();`.
- **`MaxArgumentLength` (504)** — per-property cap with a hard backstop that clamps + logs anything that slips through.
## Why
The legacy 2D client copies each OPL property's text into a fixed ~512-char (1024-byte) buffer. A single property longer than that smashes an adjacent world object's vtable on the client heap and crashes the client. `AddChunked`/`OplTextBlock` keep multi-line content safely under the cap instead of risking one oversized `Add`.
## Docs
- `dev-docs/property-lists.md` — new "Multi-Line Free Text" deep-dive section; corrected the stale `IPropertyList` listing.
- `dev-docs/claude-skills/modernuo-property-lists.md` — condensed pattern + anti-pattern.
## Tests
9 tests pass (`OplTextBlockTests`, `ObjectPropertyListSpanAddTests`): line joining, empty-line skipping, no-line no-op, zero-alloc interpolation, and long-content chunking staying under `MaxArgumentLength`. Full `UOContent` build is green, confirming dropping `Add(string)` breaks no call sites.
## Summary
Consolidates the duplicated inline AOS attribute → `ObjectPropertyList` emission that each item base copy-pastes into per-family `GetProperties(IPropertyList)` methods, mirroring the existing `AosSkillBonuses.GetProperties` precedent.
### Per-family `GetProperties(IPropertyList)`
- **`AosAttributes`** — the 24 common attributes in canonical cliloc-ascending order, with optional `damageBonus` / `hitChanceBonus` / `luckBonus` params so item-computed bonuses (e.g. `GetDamageBonus()`) stay out of the family type.
- **`AosWeaponAttributes`** — `UseBestSkill`, the `Hit*` block (1060416–1060430), `MageWeapon` (`30 - prop`), `SelfRepair`.
- **`AosArmorAttributes`** — `MageArmor`, `SelfRepair` (the always-direct members; `LowerStatReq`/`DurabilityBonus` stay inline since they're item-computed in armor but container-direct in clothing).
### Rewired all 6 `AosAttributes`-emitting item bases
`BaseJewel`, `BaseArmor`, `BaseClothing`, `BaseWeapon`, `BaseTalisman`, `Spellbook` now call the family methods instead of inlining the chain. Net: large dedup in `BaseWeapon`/`BaseArmor`/`BaseClothing`/`BaseTalisman`/`Spellbook`.
## Behavior change: tooltip line **order** (set preserved)
This is **not** a pure no-op refactor, and that's unavoidable. Today the families are emitted **interleaved in cliloc order**, and the relative order differs per item class — e.g. `BonusDex` (1060409) is emitted early in `BaseArmor` but **after** the `Hit*` block in `BaseWeapon`. No single emission order reproduces every class byte-for-byte, so consolidating into contiguous per-family blocks necessarily **de-interleaves**: lines regroup **specific → general** (family-specific, then common `AosAttributes`).
- The **set** of emitted `(cliloc, argument)` lines per item is preserved **exactly** — nothing dropped, added, or value-changed.
- Only the **order** of lines within a tooltip changes for `BaseArmor` / `BaseWeapon` / `BaseClothing`. `BaseJewel` / `BaseTalisman` / `Spellbook` were already canonical, so those are byte-identical.
## Tests
- **Golden set-invariance tests** per item base (`BaseArmor/Clothing/Jewel/Weapon/Talisman/Spellbook PropertiesTests`) — each was written to pass against current `main` **before** the rewire (locking the emitted-line set), then confirmed still passing after, proving no line is lost/added/changed.
- Family-level unit tests for each `GetProperties` (canonical order, computed-bonus folding, the `AosArmorAttributes` exclusions).
- `dotnet build` clean; full `UOContent.Tests` green. (Pre-existing `AccountPacket`/`GumpPacket`/`MobilePacket`/`ClientEnumerator` golden-test failures reproduce on unmodified `main` and are unrelated to this change.)
## Summary
`Rectangle3DConverter.Write` corrupts a rectangle's Z range for certain bounds. The omit-z guard was:
```csharp
var writeZ = value.Start.Z is > sbyte.MinValue and < sbyte.MaxValue
|| value.End.Z is > sbyte.MinValue and < sbyte.MaxValue;
```
This drops `z1`/`z2` whenever **both** Z bounds sit at/outside the sbyte extremes — but `Read` reconstructs absent Z as exactly `z1 = -128, z2 = 127` (depth 255). So any rectangle that trips the omit condition without *being* that sentinel round-trips to depth 255 and is corrupted.
The concrete case: a **homeRange-style** bound `Start.Z = -128, End.Z = 128` (depth 256, the full vertical range used by spawners) gets `z1`/`z2` omitted on write, then reads back as depth **255** — silently losing the top z-level on every re-serialize.
(Spotted while working on #2505; spawners now prefer the `homeRange` form so most square bounds avoid this path, but any non-square `spawnBounds` or other `Rectangle3D` JSON is affected.)
## Fix
Omit Z only for the exact sentinel `Read` produces (`z1 == -128 && z2 == 127`); write it for anything else:
```csharp
var writeZ = value.Start.Z != sbyte.MinValue || value.End.Z != sbyte.MaxValue;
```
`Read` is unchanged, so existing `regions.json` rectangles that omit Z keep loading identically.
## Tests
New `Rectangle3DConverterTests`: round-trips the homeRange depth-256 case, the depth-255 sentinel, and ordinary/edge z values; asserts the sentinel omits Z while homeRange bounds write it. Server.Tests 717/717, UOContent.Tests 487/487.
## Summary
Removes the dated `DynamicJson` JSON helper and migrates spawner JSON (de)serialization to a polymorphic `record SpawnerDto` hierarchy. `DynamicJson` was the last remaining consumer (regions moved off it in #1400).
The key correctness improvement: **System.Text.Json deserializes plain DTO records, never a live `Item`.** Previously, deserializing directly into an `Item` meant STJ constructed world-registered objects *before* the data was validated — a malformed/hand-edited spawn file could leave orphaned spawner Items in the world save. Now a parse failure is GC-only; `dto.ToSpawner()` constructs the spawner only from a fully-validated DTO and self-cleans on failure.
## What changed
- **New:** `SpawnerDto` (abstract) + `SpawnerDataDto` / `RegionSpawnerDto` / `ProximitySpawnerDto`, each marked with a reusable `[JsonDiscoverableType]` opt-in attribute. Auto-discovered at the `Configure` phase — no manual registration list (avoids the regions `Register<T>()` footgun), open to custom spawner subtypes.
- **Symmetric mapping:** `BaseSpawner.ToDto()` (export) ⇄ `SpawnerDto.ToSpawner()` (import). `ToSpawner()` deletes-and-rethrows on any failure, so the importer can never orphan an Item.
- **`SpawnerJsonSerializer`** wires `$type` polymorphism on the `SpawnerDto` root with loud collision/constructibility validation.
- **Import/export commands** rewired to the typed DTO path (reflection `FindTypeByName`/`CreateInstance` removed).
- **Data migration:** the 109 `Distribution/Data/Spawns/**` files moved from `"type"`→`"$type"` and legacy `homeRange`→`spawnBounds`. The runtime still *reads* legacy `homeRange` for external files. The `homeRange→spawnBounds` formula is proven equivalent to the runtime conversion (`BoundsEquivalenceTests`, hr=0/1/3/7).
- **Deleted:** `Projects/Server/Json/DynamicJson.cs`.
Sparse export output matches the legacy `ToJson` (nullable DTO properties + `WhenWritingNull`). Binary world-save serialization is untouched.
## Tests
- DTO round-trip per spawner type; sparse-default omission; legacy `homeRange` read; export/import file round-trip.
- `Import_MalformedFile_LeaksNoWorldItems` — proves a mid-array parse failure constructs zero world Items.
- `AllSpawnFilesLoadTests` — every migrated spawn file deserializes and builds.
- Duplicate-discriminator validation.
- UOContent.Tests 485/485, Server.Tests 710/710, build clean.
## Follow-up (not in this PR)
`Rectangle3DConverter.Write` (in `Projects/Server/`) omits `z1/z2` when `Start.Z == -128`, so a future server-side export of a `homeRange`-style spawner round-trips depth 256→255. Pre-existing and out of scope here (Server change); the migrated data reads correctly. Worth a separate small converter PR.
## Problem
A creature equipped with two items that resolve to the **same layer** can crash the legacy EA 2D client (use-after-free). Equipment `Layer` comes from tiledata (`Layer = (Layer)ItemData.Quality`), so a **two-handed weapon and a shield both resolve to `Layer.TwoHanded`**. `Mobile.FindItemOnLayer` even documents the invariant: *"We only allow 1 item per layer. Its an implicit contract."*
## Root cause
- `SendMobileIncoming` (0x78) **dedupes by layer** (the `layers` span) and sends only the first item per slot — so a static creature is fine.
- But the per-item **`SendEquipUpdate` (0x2E)** and the item **OPL** sends do **not** dedupe. On any equip/property delta (`Item.ProcessDelta`) they fire per item and leak the second same-layer item on its own.
- The client then holds two items on one equipment slot; when that slot is torn down (e.g. a large group of such creatures and the player runs out of range → mass remove) the legacy 2D client frees one and dereferences it → UAF. ClassicUO bounds-checks and is unaffected, but the server is emitting an invalid, self-contradictory stream either way.
## Fix
Make per-item equip/OPL sends honor the same first-item-per-layer rule `SendMobileIncoming` already uses:
- `Item.IsDupedEquipLayer()` — `m_Parent is Mobile m && m.FindItemOnLayer(m_Layer) != this` (reuses the existing helper; true when an earlier item already holds this items layer).
- `Item.ProcessDelta`: early-return before the per-client loop for a duped equipped item (skips the EquipUpdate and OPL to everyone).
- `Mobile` lift-reject re-show: skip the EquipUpdate and OPL for the dupe.
No behavioral change for valid equipment (distinct layers → never duped). For the invalid duped-layer case the second item was already omitted by the 0x78 packet; this just stops it leaking back via the per-item paths.
## Summary
House customization rejected ~40% of components — all classic/base tiles such as **sandstone** — for non-staff players. The pieces appeared briefly in the editor, then vanished before commit; only staff (GM+) could add them.
## Root cause
A data-convention mismatch introduced when housing.bin support was added (#2329).
- The OSI `housing.bin` encodes pre-AOS base pieces with the client **T2A** feature bit (`0x1`).
- `walls.txt` (and RunUO) encode the same pieces as `FeatureMask = 0` (always valid).
- `ComponentVerification.CheckValidity` validates against `ExpansionInfo.HousingFlags`, whose enum has no `0x1` bit, so base pieces failed `(HousingFlags & 0x1) != 0`.
- `HouseFoundation.Designer_Build` only enforces `ValidPiece` for `AccessLevel < GameMaster`, so staff bypassed validation while players had placements rejected — the server re-sends the design state and the client rebuilds the house from it, erasing the just-placed piece.
This only affects servers loading `housing.bin` from a UOP client; the old txt-only path (pre-#2329, like RunUO) was unaffected because base pieces are `0` there.
## Fix
Normalize the `housing.bin` feature mask to the housing-tier bits on load (`& HousingFlags.HousingEJ`). Base pieces collapse to `0` (always valid, exactly as `walls.txt` encodes them); `AOS`/`SE`/`ML`/... pass through unchanged. Both data sources now produce an identical validity table, matching RunUO behavior. `CheckValidity` and the `val != -1` anti-cheat guard are unchanged.
## Verification
- Parsed the real `housing.bin` from a 7.0.x client: sandstone (`0x345`) loads as `0x1` → `& HousingEJ` → `0` → valid; tier pieces (`0x40` SE, etc.) pass through; unregistered tiles stay `-1` → rejected.
- Confirmed OSI's own files disagree for the same pieces: `walls.txt` base `FeatureMask = 0` vs `housing.bin` `0x1`.
- `dotnet build` clean (0 warnings, 0 errors).
## Summary
Streamlines the SE-era archery ammo auto-recovery (recovering spent arrows/bolts after a miss). The mechanic was previously spread across four unrelated trigger points and a weapon-scoped timer that was divorced from the recovery state living on `PlayerMobile`. It also contained dead code.
### Problems fixed
- **Dead `!Warmode` gate** — `OnMiss` only runs from `OnSwing`, which requires warmode to fire, so the `if (!pm.Warmode)` branch that started the recovery timer could never trigger.
- **Scattered, divorced state** — banked ammo lived on `PlayerMobile.RecoverableAmmo` while the timer lived on the weapon (`_recoveryTimerToken`), and recovery was kicked off from four different places (`OnWarmodeChanged`, `PlayerMobile.OnDamage` kill, `BaseCreature.OnDamage` kill, `OnBeforeDeath`).
- **Per-player footprint** — every `PlayerMobile` carried a `RecoverableAmmo` field even though ~99% never miss with a bow (and most are offline).
### New design — `AmmoRecovery` side table
- All state (banked ammo + one repeating timer) is keyed by player in a static dictionary, so only players who actually miss carry any state. Transient by design — this was never serialized.
- **One feed point:** `OnMiss` banks the spent ammo type and starts the player's timer.
- **One drain point:** the timer self-gates and gathers ammo into the backpack only once the archer has disengaged — **alive, out of warmode, and not running** — otherwise it retries next tick, so banked ammo is never lost while online.
- **"Not running" allows standing still _or_ walking.** The `Direction.Running` bit is stale after a player stops, so it's paired with movement recency (`LastMoveTime`); only an *actively* running archer is blocked.
- Removed the redundant scattered triggers, the dead `!Warmode` branch, `RecoverableAmmo`, `RecoverAmmo()`, and the now-empty `OnWarmodeChanged` override. `PlayerMobile.OnDelete` calls `AmmoRecovery.Forget`.
### Behavior notes
- On death, banked ammo is **no longer flushed to the corpse** — it stays banked and is recovered after resurrection once the archer settles (player keeps it rather than dropping it to looters).
- `OnHit` immediate recovery (the ~40% arrow-to-defender behavior) is **unchanged**.
## Test plan
- [x] `dotnet build Projects/UOContent/UOContent.csproj -c Release` — succeeds, 0 warnings, 0 errors.
- [ ] In-game (SE era): miss bow shots, then disengage (drop warmode + stop) and confirm the "You recover N arrows/bolts" message and backpack contents; confirm recovery does **not** fire while running and **does** while walking/standing.
Kamron Batman