feat(network): lean base pools (#2641)
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run
**Follow-on to #2639 (merged). References a local IORingGroup `1.0.13-preview.11` pack until 1.0.13 (modernuo/IORingGroup#15) is published; do not merge before that switch.** ## Summary Consumes IORingGroup's lean base pools (modernuo/IORingGroup#15): both network pools now start with one slab, grow a slab at a time with the population, and trim idle slabs back after quiet periods. - Fixes the transport's send-pool cap: previously only 1024 of the 4096 connections could get a send buffer; connection 1025 was closed at accept. - Network memory at boot drops from about 96 MB to about 10 MB at the defaults; a full 4096 logged-in connections is about 1.25 GB of base buffers plus the growth budget. - New settings: `network.initialBufferSlabs` (default 1; slabs of each pool held from boot and the trim floor) and `network.maxBufferSlabs` (default 128; divides the connection maximum into slabs, 32 connections per slab). Both are coerced with a warning; the same value feeds the ring table and the manager so they cannot drift. - The Debug-only maintenance line includes base-pool capacity and releases. - `dev-docs/server-requirements.md` rewrites the network memory story and adds the two settings. ## Pre-auth buffers Every connection starts on the transport's platform-minimum buffers (4 KB receive, 4 KB send) instead of the base pools. It is promoted to full-size buffers (64 KB receive, `network.sendBufferSize` send) when the game server verifies its account — the point where `NetState.Account` is assigned in the `GameServer_AwaitingGameServerLogin` or `GameServer_LoggedIn` state, so a verdict that lands after the parser has moved on still promotes. Nothing ever moves back. The login-server pass stays on the small buffers for its whole lifetime. Before credentials verify, nothing promotes: the 4 KB send ring is the entire pre-auth send budget, and a connection that overruns it is dropped as exhausted, exactly as the receive side drops a packet header declaring more bytes than the receive buffer can hold (new guard in `HandlePacket`; it also closes the old 65535-byte edge on 64 KB buffers). The stock login sequence sends under 2 KB. After credentials verify, the send path promotes on demand if it ever needs to (unbudgeted, outside the memory ceiling and the shrink bookkeeping — promotion is not growth), and the oversize-packet guard waits on a pending receive promotion or retries a stalled one once for a verified account before disconnecting. A completion that fills the receive buffer arms no receive, so `HandleReceive` now calls `RingSocket.ResumeReceive()` after the parse loop — at 4 KB a burst of small packets fills the buffer in one completion. Net effect: a flood of unauthenticated connections tops out at about 32 MB across the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and the base pools used by logged-in players are separate), and never allocates a base-pool slab. Platform note: on Windows Server 2012 R2 / 2016 the transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is 64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. The server logs the effective sizes at startup. ## Testing Server.Tests (905) and UOContent.Tests green on the preview pack (one pre-existing `FamiliarAITests` failure from #2644 reproduces on `main`, tracked separately). New tests cover both coercions, that the ring's registration table equals `RequiredRegisteredBuffers` for the configured values, promotion on game-server auth and on a late account, no promotion on the login server, pre-credential overrun ending in exhaustion, post-credential on-demand promotion, the oversize-packet guard through loopback (error, wait, retry with an account, and a promotion made pending mid-parse), and receiving again after a burst fills the initial buffer.
This commit is contained in:
parent
9d9e672a09
commit
24bcfee554
6 changed files with 682 additions and 22 deletions
|
|
@ -68,13 +68,34 @@ Optional systems can add substantially more. The pathfinding prebake
|
|||
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
|
||||
small hosts.
|
||||
|
||||
Network buffers are 64 KB receive plus a configurable send buffer per connection. Send memory is
|
||||
`network.sendBufferSize` at rest and can grow to `network.sendBufferMaxSize` under load. Shared
|
||||
send-buffer tier memory is capped by `network.sendBufferGrowthBudget`, and growth is refused when
|
||||
process memory exceeds `network.memoryCeilingPercent` of available memory. The worst case is the
|
||||
base send-buffer size times the connection count, plus the shared growth budget: at the defaults,
|
||||
100 players is roughly 32 MB at rest, and the growth budget can add up to another 256 MB under
|
||||
load.
|
||||
Network buffers come from four pools that grow and shrink with the population rather than being
|
||||
sized for a full shard. A connection that has not yet presented valid credentials holds a 4 KB
|
||||
receive and a 4 KB send buffer (the platform's page size). On Windows Server 2012 R2 / 2016 the
|
||||
transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is
|
||||
64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. Everything the server
|
||||
sends before that must fit in that ring — a connection that overruns it is dropped; the stock login
|
||||
sequence uses under 2 KB. Otherwise, when the game server verifies the account the connection is
|
||||
promoted to a 64 KB receive buffer and a `network.sendBufferSize` send buffer from the base pools,
|
||||
and nothing ever moves back. A flood of unauthenticated connections tops out at about 32 MB across
|
||||
the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and
|
||||
the base pools used by logged-in players are separate), and never allocates a base-pool slab.
|
||||
At boot the network holds `network.initialBufferSlabs` slab(s) of each pool — at the defaults one
|
||||
2 MB receive slab, one 8 MB send slab and two 128 KB pre-auth slabs, about 10 MB — and allocates
|
||||
another slab only when the population needs one. Each slab covers 32 connections at the
|
||||
4096-connection maximum. After 15 quiet minutes idle slabs are trimmed back towards current usage,
|
||||
never past the last 15 minutes' peak, at one slab per pool per minute and never below
|
||||
`network.initialBufferSlabs`. Only the newest slab is trimmed, and buffers are handed out from the
|
||||
oldest slab first, so ordinary churn empties the newest slabs; a shard that drops from 4096 players
|
||||
to a handful takes about two hours to shrink fully, longer if a long-lived connection still holds a
|
||||
buffer in a newer slab.
|
||||
|
||||
Send memory per authenticated connection is `network.sendBufferSize` at rest and can grow to
|
||||
`network.sendBufferMaxSize` under load. Shared send-buffer tier memory is capped by
|
||||
`network.sendBufferGrowthBudget`, and growth is refused when process memory exceeds
|
||||
`network.memoryCeilingPercent` of available memory. The worst case is the receive and base
|
||||
send-buffer sizes times the number of logged-in connections, plus the shared growth budget: a full
|
||||
4096 logged-in connections is roughly 1.25 GB of base buffers, and the growth budget can add up to
|
||||
another 256 MB.
|
||||
|
||||
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
|
||||
Server GC on a 2-core box.
|
||||
|
|
@ -118,6 +139,8 @@ See the README for the full supported list. Two things are worth calling out:
|
|||
| `network.sendBufferMaxSize` | 2 MB (`2097152`) | Ceiling a single connection's send buffer can grow to under load. Lower it on memory-constrained hosts; raise it if slow clients are disconnected with "send buffer exhausted". |
|
||||
| `network.sendBufferGrowthBudget` | 256 MB (`268435456`) | Cap on the shared memory the larger send-buffer tiers may use. Lower it on memory-constrained hosts. |
|
||||
| `network.memoryCeilingPercent` | 80% | Refuse send-buffer growth once the process is above this share of available memory; 0 turns the check off. |
|
||||
| `network.initialBufferSlabs` | `1` | Slabs of each base pool held from boot, and the floor the trim never goes below. Raise it on a large shard to pre-warm the pools instead of paying for a slab as the population climbs. |
|
||||
| `network.maxBufferSlabs` | `128` | Divides the connection maximum into base-pool slabs: a slab holds `MaxConnections / maxBufferSlabs` connections, 32 at the default. Raise it for finer slabs on a small host (the slab floor is 16 buffers); lowering it makes each slab, and the boot allocation, larger. It is not a connection or memory cap — both pools still reach the connection maximum. |
|
||||
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
|
||||
|
||||
## Am I undersized?
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue