Follow-up to #2721. It fixes how the remaining tests behave and changes no production code.
## CI
- Both `dotnet test` steps pass `--filter "Category!=Manual"`. Before this, Manual-tagged tests ran on CI anyway.
- The one surviving movement-throttle latency test is the only coverage of burst forgiveness over a full, wrapped history. It loses its Manual tag, so it runs on CI.
## Tests that passed on CI without testing anything
- **Corpse and throwing tests.** They relied on tiledata for the weapon `Layer`, so on CI the equip silently failed and they asserted against empty lists. They now set `Layer` explicitly and assert the equip took effect. `TestThrown` also pins its own strength and range values, so a shard retuning SoulGlaive no longer breaks them.
- **Healthbar poison rows.** Server.Tests registers no poisons, so every row tested `null`. They now use real poison levels.
- **Account packet tests.** The mock accounts reported `Length` 0, so the character-list tests only ever encoded empty slots. They now encode the real characters.
- **Pathing and AI tests that only mean something on real Trammel/Felucca terrain** are now guarded with `TileDataRequirement.SkipIfMissing()`: Britain desk and clutter scenes, house footprints (no MultiData on CI) and fingerprints. Tests that are valid on flat blank terrain stay unguarded. (A synthetic tiledata fixture to run these on CI is a planned follow-up.)
- **Assertions that couldn't fail now check their claim.** Affected tests include the `DefaultWalker` cache use, cache on/off parity requiring a path, the recorder capability flags, the idle-pet gate, the monthly DST recurrence, the AdvancedSearch `Record.Exception` wrappers, `CanFitItem` with no branch-guarded assert, the Firewall callback, and `Import_NoMap`. The Geometry Parse asserts also now use the same input as their TryParse.
## Shared state
Every test that changes process-wide state now restores it, even when an assert fails:
- **The clock.** A new `TimerWheelScope` saves and restores `Core._tickCount`, `Core._now` and the timer wheel. `SimulatedClockScope` covers decay tests.
- **StepCache settings and caches**, via `StepCacheStateScope`.
- **Network state.** The ManualAllowlist, LoginAllowlist, firewall and BanChannel reporters are restored. Global packet handlers are now registered per test and removed afterwards, so they no longer stay registered for later tests.
- **Data and config tables.** `ArchiveJournal` statics, `TileData.ItemTable`, `ObjectPropertyList.Enabled`, `Core.Expansion` and the localization table are restored. One `MobileStatusVersion` restore was resetting a different expansion than the one the test changed.
- **Entities.** Tests with fixed serials now delete what they create. Fixed mobile serials moved to `World.NewMobile`, because deleting a hard-coded serial could unregister a live entity.
- **Collection race.** `SpiralScanTests` no longer clears the spawn cache from a separate collection that could run in parallel with the cache-manager tests.
- **Shared statics.** Classes reading shared static allowlists joined the sequential collection.
## Duplication
New shared helpers replace copy-pasted harness code:
- **Server.Tests:** `PacketAssert.Sends`, used by about 130 packet tests; `TestClientHarness`, `PersistencePipelineScope`, `NetTestHelpers`, `EntityCleanup` and one `TestMulti`.
- **UOContent.Tests:** `TimerWheelScope`, which also holds one wraparound-safe `RunFor`/`RunUntil`; `AiTestStubs`, which merges six walker copies and five identical creature stubs; `MobileTestHarness`, `CommandTestHarness` and `TempDirectory`.
Dead oracle packet classes are removed.
## Cost
| Test | Before | After |
|---|---|---|
| `VirtualSerialTests` | 1.57s | 2ms (starts near the wrap via reflection instead of drawing ~17.9M serials) |
| Event scheduler recurrence loop | 473ms | 1ms |
| CrowdSec drain | 517ms | 16ms (event-driven instead of a fixed 500ms wait) |
| AuthId tests | 248ms | 99ms (SHA2 except the two GameLogin tests) |
| Container cap tests | 5,000 items | 3,449 / 3,277 (one past each protocol cap) |
| Bulk removal | 100k children | 20k children |
| OPL hash distribution | 60k builds | 10k builds |
## Shard-tuning decoupling
- AI assertions read `ReacquireDelay` and `FollowMoveSpeed` instead of hard-coding production tuning.
- The guard friendly-fire test uses an `AI_Mage` stub instead of WhiteWyrm.
- Argon2 assertions pin the `argon2id` variant and `v=19` but read the cost parameters from the configured hasher.
## Tests
| Run | Server.Tests | UOContent.Tests |
|---|---|---|
| With client files | 812 pass | 1,123 pass |
| Without client files (CI simulation) | 800 pass, 12 skip | 1,021 pass, 102 skip |
- The UOContent fixture reads `MODERNUO_TEST_DATA_DIR`; Server.Tests reads `MODERNUO_CLIENT_PATH`.
- The 15 new UOContent skips are the map-dependent guards above.
- Total run time with client files is about 5s, down from about 8s.
## Problem
A connection's send buffer is a fixed 256 KB. A burst of world traffic (a crowded area, a mass spawn, a war) that outruns the client's acknowledgements fills it, `NetState.Send()` reports "send buffer exhausted", and the player is disconnected. Raising the size for everyone multiplies the per-connection footprint (4096 × 256 KB is already 1 GB at full occupancy, page-locked on Windows).
## What changes
- **Growth.** When a packet does not fit (the write span is too small, the packet is larger than the span, or compression returns 0), `Send()` asks the transport to grow the buffer to the next power-of-two tier and retries, up to `network.sendBufferMaxSize` (2 MB). Compression retries once per tier since its output size is not known in advance, including when the buffer is completely full. Only when growth is refused does the existing exhaustion disconnect run. The success path is unchanged.
- **Memory ceiling.** Growth is refused (with a once-a-minute warning) when the process working set exceeds `network.memoryCeilingPercent` (80) of the memory available to the process (container-aware; `0` turns the check off). The figure is sampled at startup and refreshed each maintenance tick.
- **Shrink.** A grown socket returns to the base buffer once it is drained and 30 s have passed since its last growth, attempted from the `DataSent` handler and from the 5 s alive sweep.
- **Retention.** Every minute a timer calls the transport's `Maintain()`, which trims idle tier slabs down to the peak concurrent usage of the last 15 minutes, so recurring bursts reuse buffers without allocation while rare ones give the memory back. The line logs at Debug, and only when capacity, usage, or the floor changed or a growth was refused (budget, at max, or ceiling), so an idle shard logs nothing.
- **Budget.** `network.sendBufferGrowthBudget` (256 MB) caps the tier pools' capacity; a positive value below one tier slab is raised with a warning, a negative one is clamped to 0 (growth off). Worst case is base × connections plus the budget.
- Settings are coerced with accurate warnings (power of two, minimum, 256 MB transport ceiling). `[dumpnetstates` gains the send buffer size. `dev-docs/server-requirements.md` describes the new memory story.
## Tests
`NetStateSendBufferTests` (real loopback sockets): growth instead of disconnect with a byte-exact stream, compressed growth against the compressor's own output, the grow-then-copy path, growth with a send genuinely in flight, refusal past the maximum, refusal under the ceiling, refusal on a closing socket, shrink after the hold (direct and through the alive sweep), and the setting coercions. Server.Tests 891 passed, UOContent.Tests 1052 passed against the published 1.0.12.
Reviewed per task, whole-branch, and adversarially by a second model (twice, the second time jointly with the transport branch); all findings addressed.
## Problem
When a client's send buffer fills, the `send buffer exhausted` warning repeats for every packet, every tick, until the NetState is finally disposed. Before the io_uring transport an overflow produced one message plus the disconnect line.
## Root cause
Since #2315, `NetState.Disconnect()` only queues. The NetState keeps running, the Mobile stays attached, and in `Slice()` the queued disconnect becomes `RingSocket.Disconnect()`, which sees buffered or in-flight sends and merely sets `DisconnectPending` while the transport drains. Nothing stopped game logic from writing into that buffer afterwards, so:
- every broadcast to that player still reached `Send()`, hit the full buffer, and re-reported exhaustion (made visible by #2551);
- refills kept `ReadableBytes` above zero, so the graceful drain could never finish, and `DataSent` completions kept pushing the alive check out. A slow-but-acking client could keep a "disconnected" session attached indefinitely.
Two more sources of the same warning surfaced during the analysis: `Dispose()` sets `_running = false` before nulling `Mobile.NetState`, and the setter's bank-close / target-cancel packets then reported `0 writable`; and when the socket takes the immediate-close branch (nothing in flight) `Connected` drops without `DisconnectPending`, leaving a one-tick window that also re-reported.
## Changes
- `CannotSendPackets()` refuses once the socket is `DisconnectPending` or no longer `Connected`. Sends between `Disconnect()` and the `Slice()` handoff are still delivered (kicks with a message, the play-server ack).
- `SendBufferExhausted()` reports once per disconnect and keeps the first reason.
- `Send()` is silent while closing instead of reporting exhaustion for a socket that is going away.
- `_nextAliveCheck` is seeded from a real tick (the zero default suppressed the alive sweep on hosts whose counter starts negative).
- `CancelAllTrades()` had its null guard inverted since 547c2ea0f (#2603) and never cancelled anything.
## Does the gate cut off the graceful flush?
No. The gate only refuses writes made after `Slice()` has handed the disconnect to the socket. Everything committed before that point is drained by the transport, which then sends FIN. This matches the pre-io_uring lifecycle: `Disconnect()` cleared `_running` at once, and the next `Slice()` made its final `Flush()` and then closed the socket in `Dispose()`. In both worlds the send window after `Disconnect()` ends at the next network slice; the old one made a single flush attempt, the new one drains everything buffered.
Checked flows:
- **Login gateway.** `PlayServer` sends the 0x8C ack, and the parser queues `Disconnect()` in the same `HandleReceive` call. Both happen before the handoff, so the ack is delivered and FIN follows. ClassicUO's `HandleRelayServerPacket` disconnects and opens a fresh connection before sending the seed and second login, so the game login is a new NetState. The `LoginServer_ServerSelectAck` "CUO/Orion do not reconnect" fallback (#489, 2021) resets the parser state and returns without parsing or replying, and no further receive is posted once the disconnect is pending, so it sends nothing either way. Orion likewise opens a separate game socket before closing the login one.
- **Login rejections, character create/select/delete errors, duplicate-packet guards.** Each sends its rejection first and calls `Disconnect()` in the same handler.
- **Kicks and bans** (`[kick`, `[ban`, AdminGump, ClientGump, ClientVerification, AssistantHandler, lockdown). The message is sent first; delayed variants fire `Disconnect()` from a timer with nothing sent afterwards.
- **Main loop order.** Timers run before `NetState.Slice()`, packet handlers run inside it before the flush, and `LoopContext` tasks run after it; in every case a send that precedes `Disconnect()` reaches the buffer before the next handoff.
`Send_BeforeDisconnect_IsDeliveredThenPeerSeesEof` and `Send_LargeBeforeDisconnect_IsFullyDrainedThenPeerSeesEof` read the bytes back from the peer socket after the handoff (the latter 192 KB across several send completions, past the loopback kernel buffers) and then wait for the FIN, proving delivery and clean close with the gate in place.
## Tests
Seven new tests in `NetStateDisconnectTests` over real loopback sockets: force-close after the drain deadline, delivery then EOF for small and multi-completion sends before `Disconnect()`, send dropped after the handoff, send dropped after an immediate close, exhaustion reported once with the first reason kept, and trades cancelled on disconnect. `MockAccount` promoted to a shared test helper. Server.Tests 876 passed, UOContent.Tests 1048 passed against the published 1.0.11.
## Drain deadline and IORingGroup 1.0.11
A socket handed a disconnect drains what is buffered and closes once the peer has acknowledged it. Send completions keep `NextActivityCheck` moving, so a slow but acking peer could hold a closing socket open indefinitely. A deadline (`DrainTimeoutMs`, 10 s) is now armed at the handoff, or on first sight of a transport-initiated drain in `CheckAlive`, and force-closes when it passes, independent of the inactivity check.
That force-close is only safe with IORingGroup 1.0.11 (modernuo/IORingGroup#12), which this PR bumps to. Before it, `DisconnectImmediate` released pooled buffers while recv/send operations could still be in flight, a failed send stranded the socket forever, and a recv completion could be delivered after its buffer was released. 1.0.11 retires every outstanding operation before release, aborts on a failed send, and holds buffers until the pass after the `Disconnected` event.
Kamron Batman