feat(network): lean base pools (#2641)
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run

**Follow-on to #2639 (merged). References a local IORingGroup `1.0.13-preview.11` pack until 1.0.13 (modernuo/IORingGroup#15) is published; do not merge before that switch.**

## Summary

Consumes IORingGroup's lean base pools (modernuo/IORingGroup#15): both network pools now start with one slab, grow a slab at a time with the population, and trim idle slabs back after quiet periods.

- Fixes the transport's send-pool cap: previously only 1024 of the 4096 connections could get a send buffer; connection 1025 was closed at accept.
- Network memory at boot drops from about 96 MB to about 10 MB at the defaults; a full 4096 logged-in connections is about 1.25 GB of base buffers plus the growth budget.
- New settings: `network.initialBufferSlabs` (default 1; slabs of each pool held from boot and the trim floor) and `network.maxBufferSlabs` (default 128; divides the connection maximum into slabs, 32 connections per slab). Both are coerced with a warning; the same value feeds the ring table and the manager so they cannot drift.
- The Debug-only maintenance line includes base-pool capacity and releases.
- `dev-docs/server-requirements.md` rewrites the network memory story and adds the two settings.

## Pre-auth buffers

Every connection starts on the transport's platform-minimum buffers (4 KB receive, 4 KB send) instead of the base pools. It is promoted to full-size buffers (64 KB receive, `network.sendBufferSize` send) when the game server verifies its account — the point where `NetState.Account` is assigned in the `GameServer_AwaitingGameServerLogin` or `GameServer_LoggedIn` state, so a verdict that lands after the parser has moved on still promotes. Nothing ever moves back. The login-server pass stays on the small buffers for its whole lifetime.

Before credentials verify, nothing promotes: the 4 KB send ring is the entire pre-auth send budget, and a connection that overruns it is dropped as exhausted, exactly as the receive side drops a packet header declaring more bytes than the receive buffer can hold (new guard in `HandlePacket`; it also closes the old 65535-byte edge on 64 KB buffers). The stock login sequence sends under 2 KB. After credentials verify, the send path promotes on demand if it ever needs to (unbudgeted, outside the memory ceiling and the shrink bookkeeping — promotion is not growth), and the oversize-packet guard waits on a pending receive promotion or retries a stalled one once for a verified account before disconnecting. A completion that fills the receive buffer arms no receive, so `HandleReceive` now calls `RingSocket.ResumeReceive()` after the parse loop — at 4 KB a burst of small packets fills the buffer in one completion.

Net effect: a flood of unauthenticated connections tops out at about 32 MB across the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and the base pools used by logged-in players are separate), and never allocates a base-pool slab.

Platform note: on Windows Server 2012 R2 / 2016 the transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is 64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. The server logs the effective sizes at startup.

## Testing

Server.Tests (905) and UOContent.Tests green on the preview pack (one pre-existing `FamiliarAITests` failure from #2644 reproduces on `main`, tracked separately). New tests cover both coercions, that the ring's registration table equals `RequiredRegisteredBuffers` for the configured values, promotion on game-server auth and on a late account, no promotion on the login server, pre-credential overrun ending in exhaustion, post-credential on-demand promotion, the oversize-packet guard through loopback (error, wait, retry with an account, and a promotion made pending mid-parse), and receiving again after a burst fills the initial buffer.
This commit is contained in:
Kamron Batman 2026-09-20 00:24:15 -07:00 committed by GitHub
parent 9d9e672a09
commit 24bcfee554
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 682 additions and 22 deletions

View file

@ -68,13 +68,34 @@ Optional systems can add substantially more. The pathfinding prebake
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
small hosts.
Network buffers are 64 KB receive plus a configurable send buffer per connection. Send memory is
`network.sendBufferSize` at rest and can grow to `network.sendBufferMaxSize` under load. Shared
send-buffer tier memory is capped by `network.sendBufferGrowthBudget`, and growth is refused when
process memory exceeds `network.memoryCeilingPercent` of available memory. The worst case is the
base send-buffer size times the connection count, plus the shared growth budget: at the defaults,
100 players is roughly 32 MB at rest, and the growth budget can add up to another 256 MB under
load.
Network buffers come from four pools that grow and shrink with the population rather than being
sized for a full shard. A connection that has not yet presented valid credentials holds a 4 KB
receive and a 4 KB send buffer (the platform's page size). On Windows Server 2012 R2 / 2016 the
transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is
64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. Everything the server
sends before that must fit in that ring — a connection that overruns it is dropped; the stock login
sequence uses under 2 KB. Otherwise, when the game server verifies the account the connection is
promoted to a 64 KB receive buffer and a `network.sendBufferSize` send buffer from the base pools,
and nothing ever moves back. A flood of unauthenticated connections tops out at about 32 MB across
the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and
the base pools used by logged-in players are separate), and never allocates a base-pool slab.
At boot the network holds `network.initialBufferSlabs` slab(s) of each pool — at the defaults one
2 MB receive slab, one 8 MB send slab and two 128 KB pre-auth slabs, about 10 MB — and allocates
another slab only when the population needs one. Each slab covers 32 connections at the
4096-connection maximum. After 15 quiet minutes idle slabs are trimmed back towards current usage,
never past the last 15 minutes' peak, at one slab per pool per minute and never below
`network.initialBufferSlabs`. Only the newest slab is trimmed, and buffers are handed out from the
oldest slab first, so ordinary churn empties the newest slabs; a shard that drops from 4096 players
to a handful takes about two hours to shrink fully, longer if a long-lived connection still holds a
buffer in a newer slab.
Send memory per authenticated connection is `network.sendBufferSize` at rest and can grow to
`network.sendBufferMaxSize` under load. Shared send-buffer tier memory is capped by
`network.sendBufferGrowthBudget`, and growth is refused when process memory exceeds
`network.memoryCeilingPercent` of available memory. The worst case is the receive and base
send-buffer sizes times the number of logged-in connections, plus the shared growth budget: a full
4096 logged-in connections is roughly 1.25 GB of base buffers, and the growth budget can add up to
another 256 MB.
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
Server GC on a 2-core box.
@ -118,6 +139,8 @@ See the README for the full supported list. Two things are worth calling out:
| `network.sendBufferMaxSize` | 2 MB (`2097152`) | Ceiling a single connection's send buffer can grow to under load. Lower it on memory-constrained hosts; raise it if slow clients are disconnected with "send buffer exhausted". |
| `network.sendBufferGrowthBudget` | 256 MB (`268435456`) | Cap on the shared memory the larger send-buffer tiers may use. Lower it on memory-constrained hosts. |
| `network.memoryCeilingPercent` | 80% | Refuse send-buffer growth once the process is above this share of available memory; 0 turns the check off. |
| `network.initialBufferSlabs` | `1` | Slabs of each base pool held from boot, and the floor the trim never goes below. Raise it on a large shard to pre-warm the pools instead of paying for a slab as the population climbs. |
| `network.maxBufferSlabs` | `128` | Divides the connection maximum into base-pool slabs: a slab holds `MaxConnections / maxBufferSlabs` connections, 32 at the default. Raise it for finer slabs on a small host (the slab floor is 16 buffers); lowering it makes each slab, and the boot allocation, larger. It is not a connection or memory cap — both pools still reach the connection maximum. |
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
## Am I undersized?