ModernUO/Projects/Server
Kamron Batman 294dcd94a0
fix: Fixes send-path backpressure: consume IORingGroup 1.0.8, stop dropping packets silently (#2551)
## Summary

Two related fixes on the outbound path:

1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it.
2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state.

## 1. Send-path stall (RIO)

RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued.

Measured on a 50ms-RTT production shard:

| | before | after |
|---|---|---|
| in-game latency, data flowing | **101–146 ms** | **48–51 ms** |
| p95 | ~135 ms | 52.8 ms |
| samples > 70 ms | 20 | **0** |

The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did.

Two things worth recording, because they explain why this went unnoticed:

- As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones.
- The same measurement at loopback RTT is **microseconds**, so local testing could never surface it.

New settings, both restart-time:

- **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`.
- **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling.

## 2. Send buffer full

`NetState.Send` had three failure modes once the buffer filled, none of them visible:

| writable | behaviour |
|---|---|
| `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect |
| `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way |
| `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** |

The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space.

`Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`.

## Testing

`NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical.

Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8.

## Notes for reviewers

- Upstream change: modernuo/IORingGroup#9.
- The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise.
- Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
2026-07-27 23:06:53 -07:00
..
Buffers feat(buffers): add :L lowercase format spec to RawInterpolatedStringHandler (#2440) 2026-05-03 18:24:57 -07:00
Client feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Collections feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) 2026-07-25 11:59:37 -07:00
Comparers fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Compression feat(pathfinding): .swb format v8 compact index (#3b) (#2471) 2026-06-07 01:22:20 -07:00
Configuration fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
Console fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
ContextMenus fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Events fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Exceptions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
GarbageCollection chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Geometry fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Items fix: Streamlines insurance. Insurance only executes when enabled. (#2550) 2026-07-26 09:47:49 -07:00
Json fix(json): Rectangle3DConverter loses a z-level on write (#2506) 2026-06-25 23:42:49 -07:00
Localization feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Maps fix: Fixes pathfinding prebake and pathfinding multi-fallthrough. (#2478) 2026-06-08 11:59:24 -07:00
Menus feat: Pre-Publish 14 Crafting (supersedes #2181, #2381) (#2476) 2026-06-07 20:27:22 -07:00
Migrations refactor: decompose mobile/corpse hair, delete VirtualHairInfo, fix removal serial (#2462) (#2463) 2026-06-06 14:33:28 -07:00
Mobiles fix: Streamlines insurance. Insurance only executes when enabled. (#2550) 2026-07-26 09:47:49 -07:00
Network fix: Fixes send-path backpressure: consume IORingGroup 1.0.8, stop dropping packets silently (#2551) 2026-07-27 23:06:53 -07:00
PropertyList fix(opl): only apply the ':#' cliloc marker to integer values (#2540) 2026-07-19 10:49:16 -07:00
Random fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Regions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serialization perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Targeting chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Text feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
TileMatrix feat(pathfinding): JSONL recorder + public bake helpers (#2449) 2026-05-06 13:06:28 -07:00
Timer feat: Adds dynamic thread idle to address CPU usage (#2370) 2026-03-13 23:53:49 -07:00
TimeZones fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Utilities feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) 2026-07-25 11:59:37 -07:00
World perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
AssemblyHandler.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Attributes.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
CityInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Commands.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Effects.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EntityFinalizationTracker.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EventLoopTasks.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
ExpansionInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
FeatureFlags.cs fix: Streamlines insurance. Insurance only executes when enabled. (#2550) 2026-07-26 09:47:49 -07:00
Guild.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
HuePicker.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
IAccount.cs fix: Fixes looking up accounts that were renamed. (#2078) 2025-01-18 17:11:43 -08:00
IEntity.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Interfaces.cs feat: Add CanSpawnMobile overload with props Z-range support. (#2293) 2025-12-27 17:01:15 -08:00
IVirtualCheckGump.cs feat: Moves gumps out of the core (#1916) 2024-08-09 19:07:32 -07:00
KeywordList.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Main.cs feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) 2026-07-25 11:59:37 -07:00
MessageType.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Module.cs fix: Fixes sending packets and sidesteps a major issue with stackalloc and PGO in .NET 8 (#1607) 2023-11-21 12:18:20 -08:00
Party.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Poison.cs feat: Refactor Poison system, implement Darkglow & Parasitic effects (#2385) 2026-03-21 21:27:22 -07:00
Prompt.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Race.cs fix: Fixes TryParse for Race (#1244) 2022-11-12 00:31:25 -08:00
SecureTrade.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serial.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Server.csproj fix: Fixes send-path backpressure: consume IORingGroup 1.0.8, stop dropping packets silently (#2551) 2026-07-27 23:06:53 -07:00
Skills.cs chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
TileData.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
TileList.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00