feat(network): lean base pools (#2641)
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run

**Follow-on to #2639 (merged). References a local IORingGroup `1.0.13-preview.11` pack until 1.0.13 (modernuo/IORingGroup#15) is published; do not merge before that switch.**

## Summary

Consumes IORingGroup's lean base pools (modernuo/IORingGroup#15): both network pools now start with one slab, grow a slab at a time with the population, and trim idle slabs back after quiet periods.

- Fixes the transport's send-pool cap: previously only 1024 of the 4096 connections could get a send buffer; connection 1025 was closed at accept.
- Network memory at boot drops from about 96 MB to about 10 MB at the defaults; a full 4096 logged-in connections is about 1.25 GB of base buffers plus the growth budget.
- New settings: `network.initialBufferSlabs` (default 1; slabs of each pool held from boot and the trim floor) and `network.maxBufferSlabs` (default 128; divides the connection maximum into slabs, 32 connections per slab). Both are coerced with a warning; the same value feeds the ring table and the manager so they cannot drift.
- The Debug-only maintenance line includes base-pool capacity and releases.
- `dev-docs/server-requirements.md` rewrites the network memory story and adds the two settings.

## Pre-auth buffers

Every connection starts on the transport's platform-minimum buffers (4 KB receive, 4 KB send) instead of the base pools. It is promoted to full-size buffers (64 KB receive, `network.sendBufferSize` send) when the game server verifies its account — the point where `NetState.Account` is assigned in the `GameServer_AwaitingGameServerLogin` or `GameServer_LoggedIn` state, so a verdict that lands after the parser has moved on still promotes. Nothing ever moves back. The login-server pass stays on the small buffers for its whole lifetime.

Before credentials verify, nothing promotes: the 4 KB send ring is the entire pre-auth send budget, and a connection that overruns it is dropped as exhausted, exactly as the receive side drops a packet header declaring more bytes than the receive buffer can hold (new guard in `HandlePacket`; it also closes the old 65535-byte edge on 64 KB buffers). The stock login sequence sends under 2 KB. After credentials verify, the send path promotes on demand if it ever needs to (unbudgeted, outside the memory ceiling and the shrink bookkeeping — promotion is not growth), and the oversize-packet guard waits on a pending receive promotion or retries a stalled one once for a verified account before disconnecting. A completion that fills the receive buffer arms no receive, so `HandleReceive` now calls `RingSocket.ResumeReceive()` after the parse loop — at 4 KB a burst of small packets fills the buffer in one completion.

Net effect: a flood of unauthenticated connections tops out at about 32 MB across the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and the base pools used by logged-in players are separate), and never allocates a base-pool slab.

Platform note: on Windows Server 2012 R2 / 2016 the transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is 64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. The server logs the effective sizes at startup.

## Testing

Server.Tests (905) and UOContent.Tests green on the preview pack (one pre-existing `FamiliarAITests` failure from #2644 reproduces on `main`, tracked separately). New tests cover both coercions, that the ring's registration table equals `RequiredRegisteredBuffers` for the configured values, promotion on game-server auth and on a late account, no promotion on the login server, pre-credential overrun ending in exhaustion, post-credential on-demand promotion, the oversize-packet guard through loopback (error, wait, retry with an account, and a promotion made pending mid-parse), and receiving again after a burst fills the initial buffer.
This commit is contained in:
Kamron Batman 2026-09-20 00:24:15 -07:00 committed by GitHub
parent 9d9e672a09
commit 24bcfee554
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 682 additions and 22 deletions

View file

@ -0,0 +1,236 @@
using System;
using System.Buffers;
using System.Diagnostics;
using System.Network;
using System.Threading;
using Server.Network;
using Xunit;
namespace Server.Tests.Network;
[Collection("Sequential Server Tests")]
public class NetStateRecvGuardTests
{
private const byte TestPacketId = 0xB1;
private static unsafe void RegisterVariableLength()
{
if (IncomingPackets.GetHandler(TestPacketId) == null)
{
IncomingPackets.Register(TestPacketId, 0, false, &NoOp);
}
}
private static void NoOp(NetState state, SpanReader reader)
{
}
// Not 0x73: NetStateSendBufferTests registers its own no-op there first-come-wins, since the
// handler table is process-global, and this test needs its own handler's side effect to fire.
private const byte TestPingPacketId = 0x72;
private static byte _lastPing;
private static unsafe void RegisterNoOpPing()
{
if (IncomingPackets.GetHandler(TestPingPacketId) == null)
{
IncomingPackets.Register(TestPingPacketId, 2, false, &RecordPing);
}
}
private static void RecordPing(NetState state, SpanReader reader) => _lastPing = reader.ReadByte();
private const byte AttachAccountPacketId = 0x71;
private static unsafe void RegisterAttachAccount()
{
if (IncomingPackets.GetHandler(AttachAccountPacketId) == null)
{
IncomingPackets.Register(AttachAccountPacketId, 3, false, &AttachAccount);
}
}
// Stands in for the 0x91 handler: the account attaches mid-parse, so the promotion is pending
// with a receive armed when the next packet in the same completion reaches the guard
private static void AttachAccount(NetState state, SpanReader reader) => state.Account = new MockAccount();
private static void SliceUntil(Func<bool> done)
{
var deadline = Stopwatch.StartNew();
while (!done() && deadline.ElapsedMilliseconds < 5000)
{
NetState.Slice();
Thread.Sleep(5);
}
Assert.True(done());
}
[SkippableFact]
public void HandleReceive_PacketLongerThanTheRecvBuffer_IsAnError()
{
RegisterVariableLength();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
// A 16-bit length cannot exceed a 64 KiB buffer, so the guard is unreachable there and
// this only ever hit the `< 3` check on a buffer that size.
Skip.If(ns._socket.RecvBuffer.PhysicalSize > ushort.MaxValue);
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
// A variable-length header claiming more than the buffer can ever hold would otherwise
// park the connection with nothing to arm a recv into
var declared = ns._socket.RecvBuffer.PhysicalSize; // one more than the buffer can ever hold
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
SliceUntil(() => ns._protocolState == NetState.ProtocolState.Error);
Assert.Contains("bad state", ns._disconnectReason);
}
finally
{
ns.Dispose();
client.Close();
}
}
[Fact]
public void HandleReceive_PacketThatFits_WaitsForTheRest()
{
RegisterVariableLength();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
var declared = ns._socket.RecvBuffer.PhysicalSize / 2;
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 3);
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
Assert.True(ns.Running);
Assert.Equal(3, ns._socket.RecvBuffer.ReadableBytes);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void HandleReceive_OversizePacket_WithAnAccount_RetriesPromotionInsteadOfErroring()
{
Skip.If(NetState.InitialRecvBufferSize == 0);
RegisterVariableLength();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
// _protocolState is still AwaitingSeed here, so the Account setter's gate does not fire
ns.Account = new MockAccount();
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
var declared = NetState.InitialRecvBufferSize; // as much as the initial buffer can ever hold
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 3);
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
Assert.True(ns.Running);
// The deferred promotion applies at this next completion; the header is then delivered
// whole to the 0xB1 no-op handler
client.Send(new byte[declared - 3]);
SliceUntil(
() => ns._socket.RecvBuffer.ReadableBytes == 0 &&
ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize
);
Assert.True(ns.Running);
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void HandleReceive_AfterABurstFillsTheInitialBuffer_KeepsReceiving()
{
Skip.If(NetState.InitialRecvBufferSize == 0);
RegisterNoOpPing();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
// One segment of pings fills the 4 KiB buffer (capacity is PhysicalSize - 1, odd, so the
// last ping straddles two completions); the parser consumes them all
var capacity = ns._socket.RecvBuffer.PhysicalSize - 1;
var pings = new byte[capacity + 1];
for (var i = 0; i < pings.Length; i += 2)
{
pings[i] = TestPingPacketId;
pings[i + 1] = (byte)(i / 2);
}
client.Send(pings);
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 0 && ns._receivedData);
Assert.True(ns.Running);
// Without a re-arm this ping never arrives
client.Send(new byte[] { TestPingPacketId, 0xEE });
SliceUntil(() => _lastPing == 0xEE);
Assert.True(ns.Running);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void HandleReceive_AccountAttachedMidParse_ThenOversizeHeader_WaitsForThePendingPromotion()
{
Skip.If(NetState.InitialRecvBufferSize == 0);
RegisterAttachAccount();
RegisterVariableLength();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
var declared = ns._socket.RecvBuffer.PhysicalSize;
// One send, one completion on loopback: the account packet promotes (deferred: a receive is
// armed), then the oversize header reaches the guard with that promotion pending. If the OS
// ever splits this into two completions instead, the first assertion block still holds — the
// header then hits the guard after the promotion already applied and simply waits on the
// 64 KiB buffer, so the test cannot flake, it just exercises the weaker path on that run.
client.Send(new byte[] { AttachAccountPacketId, 0x00, 0x00, TestPacketId, (byte)(declared >> 8), (byte)declared });
SliceUntil(() => ns.Account != null);
Assert.True(ns.Running);
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
Assert.Equal(3, ns._socket.RecvBuffer.ReadableBytes); // the header waits, not rejected
// The next completion applies the promotion and the whole packet is delivered
client.Send(new byte[declared - 3]);
SliceUntil(
() => ns._socket.RecvBuffer.ReadableBytes == 0 &&
ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize
);
Assert.True(ns.Running);
}
finally
{
ns.Dispose();
client.Close();
}
}
}

View file

@ -1,4 +1,5 @@
using System;
using System.Buffers;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.Sockets;
@ -12,9 +13,11 @@ namespace Server.Tests.Network;
[Collection("Sequential Server Tests")]
public class NetStateSendBufferTests
{
// Authentication is the 0x91 credentials check on the game server; that is where buffers promote
private static NetState CreateAuthenticatedNetState(out Socket client)
{
var ns = PacketTestUtilities.CreateTestNetState(out client);
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
ns.Account = new MockAccount();
return ns;
}
@ -26,6 +29,183 @@ public class NetStateSendBufferTests
return data;
}
private static unsafe void RegisterNoOpPing()
{
if (IncomingPackets.GetHandler(0x73) == null)
{
IncomingPackets.Register(0x73, 2, false, &NoOp);
}
}
private static void NoOp(NetState state, SpanReader reader)
{
}
private static void SliceUntil(Func<bool> done)
{
var deadline = Stopwatch.StartNew();
while (!done() && deadline.ElapsedMilliseconds < 5000)
{
NetState.Slice();
Thread.Sleep(5);
}
Assert.True(done());
}
[SkippableFact]
public void Account_InTheGameServerState_PromotesBothBuffers()
{
Skip.If(NetState.InitialSendBufferSize == 0);
RegisterNoOpPing();
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
ns.Account = new MockAccount();
// Send promotes at once; nothing was in flight so no buffer retires
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
Assert.False(ns._sendBufferGrown);
// Recv promotes at the next completion
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
client.Send(new byte[] { 0x73, 0x01 });
SliceUntil(() => ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize);
Assert.True(ns.Running);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void Account_OnTheLoginServer_KeepsTheInitialBuffers()
{
Skip.If(NetState.InitialSendBufferSize == 0);
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns._protocolState = NetState.ProtocolState.LoginServer_AwaitingLogin;
ns.Account = new MockAccount();
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void Send_BeyondTheInitialBuffer_BeforeCredentials_IsExhausted()
{
// Nothing promotes before credentials verify: the 4 KiB ring is the whole pre-auth budget
Skip.If(NetState.InitialSendBufferSize == 0);
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
ns.Send(data);
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
// A graceful server-side close half-closes and waits for the peer's own FIN; closing the
// peer here stands in for the client eventually going away, so Running can be observed.
// Once it completes the NetState is disposed and _socket goes null, so nothing below
// this point may touch it.
client.Close();
SliceUntil(() => !ns.Running);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void Send_BeyondTheInitialBuffer_WithCredentials_PromotesOnDemand()
{
// The late-verdict path from Account_AssignedAfterTheStateFlipped_StillPromotes: an account
// attached while the setter's own gate does not fire must still let the send path promote
Skip.If(NetState.InitialSendBufferSize == 0);
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns.Account = new MockAccount(); // _protocolState is still AwaitingSeed: the setter does not promote
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
ns.Send(data);
Assert.True(ns.Running);
Assert.Equal(string.Empty, ns._disconnectReason);
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
Assert.False(ns._sendBufferGrown); // promotion is not growth: nothing to shrink later
Assert.Equal(data, ReadAll(client, data.Length));
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void Send_BacklogBeforeCredentials_IsExhaustedInsteadOfPromoted()
{
Skip.If(NetState.InitialSendBufferSize == 0);
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
// No Slice() between sends: nothing drains, so a second write finds a non-empty initial buffer
var half = Pattern(NetState.InitialSendBufferSize / 2, 11);
ns.Send(half);
ns.Send(half);
ns.Send(half); // cannot fit, buffer not empty, no account: refused
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
// A graceful server-side close waits for the queued bytes to drain and then for the
// peer's own FIN; closing the peer here stands in for the client eventually going away.
// Once it completes the NetState is disposed and _socket goes null, so nothing below
// this point may touch it.
client.Close();
SliceUntil(() => !ns.Running);
}
finally
{
ns.Dispose();
client.Close();
}
}
[SkippableFact]
public void Account_AssignedAfterTheStateFlipped_StillPromotes()
{
Skip.If(NetState.InitialSendBufferSize == 0);
var ns = PacketTestUtilities.CreateTestNetState(out var client);
try
{
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
ns.Account = new MockAccount();
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
}
finally
{
ns.Dispose();
client.Close();
}
}
// A random prefix lands under the target; zeros (2 bits each) walk it up a byte at a time
private static byte[] CompressesToExactly(int compressedLength, int seed)
{
@ -448,4 +628,58 @@ public class NetStateSendBufferTests
// growth off; budget untouched
Assert.Equal(1L, NetState.CoerceSendBufferGrowthBudget(1, sendBufferSize, sendBufferSize));
}
[Fact]
public void CoerceMaxBufferSlabs_ClampsToOneSlabPerConnection()
{
var maxConnections = NetState.SocketManager.MaxSockets;
// fewer than one slab is meaningless
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(0));
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(-4));
// more slabs than connections cannot make a slab any smaller
Assert.Equal(maxConnections, NetState.CoerceMaxBufferSlabs(maxConnections * 2));
Assert.Equal(128, NetState.CoerceMaxBufferSlabs(128));
}
[Fact]
public void CoerceInitialBufferSlabs_ClampsToTheSlabCount()
{
var slabs = NetState.BasePoolSlabCount(128);
Assert.True(slabs > 1);
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(0, slabs));
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(-1, slabs));
// a pool never holds more slabs than it has
Assert.Equal(slabs, NetState.CoerceInitialBufferSlabs(slabs + 1, slabs));
Assert.Equal(4, NetState.CoerceInitialBufferSlabs(4, slabs));
}
[Fact]
public void Ring_RegisteredBufferTable_MatchesTheConfiguredSlabCount()
{
var manager = NetState.SocketManager;
var maxBufferSlabs = NetState.CoerceMaxBufferSlabs(ServerConfiguration.GetSetting("network.maxBufferSlabs", NetState.DefaultMaxBufferSlabs));
// A table smaller than this throws at manager construction. Connections start on the
// transport minimum until the game server promotes them, so the formula must count those too.
// Mirrors what production passes: the request, not the effective size the manager may coerce
// down to (RequiredRegisteredBuffers treats a non-zero request as a pool either way).
Assert.Equal(
RingSocketManager.RequiredRegisteredBuffers(
manager.MaxSockets,
NetState.SendBufferSize,
manager.MaxSendBufferSize,
manager.SendBufferGrowthBudget,
maxBufferSlabs,
IORingBuffer.MinimumSize,
IORingBuffer.MinimumSize
),
NetState.Ring.MaxRegisteredBuffers
);
}
}

View file

@ -29,18 +29,26 @@ namespace Server.Network;
public partial class NetState
{
// Buffer sizes
private const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
internal const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
private const int DefaultSendBufferSize = 1024 * 256; // 256KB send buffers
private const int MinSendBufferSize = 1024 * 64; // Platform allocation granularity
private const int DefaultMaxSendBufferSize = 1024 * 1024 * 2; // 2 MB
private const long DefaultSendBufferGrowthBudget = 1024L * 1024 * 256; // 256 MB
private const int DefaultMemoryCeilingPercent = 80;
private const int DefaultInitialBufferSlabs = 1; // one slab of each base pool warm at boot
internal const int DefaultMaxBufferSlabs = 128; // 32 connections per slab at MaxConnections
// Transport ceiling; larger values overflow its tier enumeration
private const int TransportMaxSendBufferSize = 1024 * 1024 * 256; // 256 MB
private const int MaxConnections = 4096; // Max concurrent connections
internal static int SendBufferSize { get; private set; }
internal static int MaxSendBufferSize { get; private set; }
// Pre-auth buffers are the smallest the platform can map; a platform whose floor is not below
// the base size (the Windows legacy mapping path) starts sockets on base
internal static int InitialRecvBufferSize { get; private set; }
internal static int InitialSendBufferSize { get; private set; }
private static long _sendBufferGrowthBudget;
private static int _memoryCeilingPercent;
@ -51,6 +59,7 @@ public partial class NetState
private static long _lastTierCapacityBytes;
private static int _lastTierInUse;
private static int _lastTierRetainFloor;
private static long _lastBaseCapacityBytes;
private static readonly Queue<NetState> _disposed = [];
private static readonly TimeSpan ConnectingSocketIdleLimit = TimeSpan.FromMilliseconds(5000); // 5 seconds
@ -141,23 +150,46 @@ public partial class NetState
// Per-connection send buffer: the lever for "send buffer exhausted" disconnects, and the
// per-connection memory ceiling.
var sendBufferSize = GetSendBufferSize();
MaxSendBufferSize = GetPowerOfTwoSetting("network.sendBufferMaxSize", DefaultMaxSendBufferSize, sendBufferSize);
SendBufferSize = GetSendBufferSize();
MaxSendBufferSize = GetPowerOfTwoSetting("network.sendBufferMaxSize", DefaultMaxSendBufferSize, SendBufferSize);
_sendBufferGrowthBudget = CoerceSendBufferGrowthBudget(
ServerConfiguration.GetOrUpdateSetting("network.sendBufferGrowthBudget", DefaultSendBufferGrowthBudget),
sendBufferSize,
SendBufferSize,
MaxSendBufferSize
);
// 0 disables the ceiling
_memoryCeilingPercent = Math.Clamp(ServerConfiguration.GetOrUpdateSetting("network.memoryCeilingPercent", DefaultMemoryCeilingPercent), 0, 100);
_availableMemoryBytes = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes;
const int maxBufferSlabs = 32;
// Divides MaxConnections into base-pool slabs; both pools still reach MaxConnections, so
// this sets slab granularity, not a connection or memory ceiling.
var maxBufferSlabs = CoerceMaxBufferSlabs(
ServerConfiguration.GetOrUpdateSetting("network.maxBufferSlabs", DefaultMaxBufferSlabs)
);
// Slabs of each base pool held from boot; the pools grow and trim from here
var initialBufferSlabs = CoerceInitialBufferSlabs(
ServerConfiguration.GetOrUpdateSetting("network.initialBufferSlabs", DefaultInitialBufferSlabs),
BasePoolSlabCount(maxBufferSlabs)
);
// Until the game server verifies credentials a connection holds the smallest buffers the
// platform can map; a flood can fill these pools but never reaches the base pools. This is a
// request: the manager raises it to the platform floor and turns the pool off if that leaves
// no room below the base size (the Windows legacy mapping path floors at 64 KiB, which can
// equal RecvBufferSize). The effective sizes are read back below once the manager knows them.
var initialBufferSize = IORingBuffer.MinimumSize;
// Both calls take the same slab count; the manager throws if the table is smaller. Sized by
// the request, not the effective size the manager may coerce down to - conservative, never small.
var ring = IORingGroup.Create(
queueSize: MaxConnections * 2,
maxConnections: MaxConnections,
maxOutstandingSends: maxOutstandingSends,
maxRegisteredBuffers: RingSocketManager.RequiredRegisteredBuffers(MaxConnections, sendBufferSize, MaxSendBufferSize, _sendBufferGrowthBudget, maxBufferSlabs)
maxRegisteredBuffers: RingSocketManager.RequiredRegisteredBuffers(
MaxConnections, SendBufferSize, MaxSendBufferSize, _sendBufferGrowthBudget, maxBufferSlabs,
initialBufferSize, initialBufferSize
)
);
// Create socket manager which handles buffer pools and socket lifecycle
@ -165,13 +197,28 @@ public partial class NetState
ring,
maxSockets: MaxConnections,
recvBufferSize: RecvBufferSize,
sendBufferSize: sendBufferSize,
initialBufferSlabs: 8,
sendBufferSize: SendBufferSize,
initialBufferSlabs: initialBufferSlabs,
maxBufferSlabs: maxBufferSlabs,
maxSendBufferSize: MaxSendBufferSize,
sendBufferGrowthBudget: _sendBufferGrowthBudget
sendBufferGrowthBudget: _sendBufferGrowthBudget,
initialRecvBufferSize: initialBufferSize,
initialSendBufferSize: initialBufferSize
);
InitialRecvBufferSize = _socketManager.InitialRecvBufferSize;
InitialSendBufferSize = _socketManager.InitialSendBufferSize;
if (InitialRecvBufferSize == 0 || InitialSendBufferSize == 0)
{
logger.Information(
"Pre-auth buffers off where the platform floor ({Minimum} bytes) leaves no room below the base size (recv {Recv}, send {Send})",
IORingBuffer.MinimumSize,
RecvBufferSize,
SendBufferSize
);
}
_maintenanceTimer = Timer.DelayCall(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1), MaintainSendBuffers);
}
@ -193,27 +240,79 @@ public partial class NetState
var stats = _socketManager.Maintain();
var changed = stats.TierCapacityBytes != _lastTierCapacityBytes ||
stats.TierInUse != _lastTierInUse ||
stats.TierRetainFloor != _lastTierRetainFloor;
stats.TierRetainFloor != _lastTierRetainFloor ||
stats.BaseCapacityBytes != _lastBaseCapacityBytes;
_lastTierCapacityBytes = stats.TierCapacityBytes;
_lastTierInUse = stats.TierInUse;
_lastTierRetainFloor = stats.TierRetainFloor;
_lastBaseCapacityBytes = stats.BaseCapacityBytes;
// Quiet unless something moved
if (changed || stats.BuffersReleased > 0 || stats.GrowthRefusals > 0 || capRefusals > 0 || ceilingRefusals > 0)
if (changed || stats.BuffersReleased > 0 || stats.BaseBuffersReleased > 0 || stats.GrowthRefusals > 0 ||
capRefusals > 0 || ceilingRefusals > 0)
{
logger.Debug(
"Send buffer tiers: {Capacity} bytes of tier capacity, {InUse} buffers in use, floor {Floor} buffers, released {Released}, refused: budget {BudgetRefusals}, at max {CapRefusals}, ceiling {CeilingRefusals}",
"Send buffer tiers: {Capacity} bytes of tier capacity, {InUse} buffers in use, floor {Floor} buffers, released {Released}, refused: budget {BudgetRefusals}, at max {CapRefusals}, ceiling {CeilingRefusals}, base {BaseCapacity} bytes, released {BaseReleased}",
stats.TierCapacityBytes,
stats.TierInUse,
stats.TierRetainFloor,
stats.BuffersReleased,
stats.GrowthRefusals,
capRefusals,
ceilingRefusals
ceilingRefusals,
stats.BaseCapacityBytes,
stats.BaseBuffersReleased
);
}
}
/// <summary>
/// Slabs each base pool is divided into, after the transport applies its minimum slab size.
/// </summary>
internal static int BasePoolSlabCount(int maxBufferSlabs) =>
RingSocketManager.BasePoolSlabCount(MaxConnections, maxBufferSlabs);
/// <summary>
/// Clamps the base-pool slab divisor. Fewer than one slab is meaningless, and more slabs than
/// connections cannot make a slab any smaller.
/// </summary>
internal static int CoerceMaxBufferSlabs(int configured)
{
var slabs = Math.Clamp(configured, 1, MaxConnections);
if (slabs != configured)
{
logger.Warning(
"network.maxBufferSlabs {Configured} is outside 1..{Maximum}; using {Adjusted}",
configured,
MaxConnections,
slabs
);
}
return slabs;
}
/// <summary>
/// Clamps the slabs of each base pool held from boot; a pool never holds more slabs than it has.
/// </summary>
internal static int CoerceInitialBufferSlabs(int configured, int slabCount)
{
var slabs = Math.Clamp(configured, 1, slabCount);
if (slabs != configured)
{
logger.Warning(
"network.initialBufferSlabs {Configured} is outside 1..{Maximum}; using {Adjusted}",
configured,
slabCount,
slabs
);
}
return slabs;
}
/// <summary>
/// Reads the configured send buffer size, coerced to a power of two of at least the platform
/// allocation granularity. IORingBuffer requires this and would otherwise throw at socket

View file

@ -227,7 +227,39 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
public IAccount Account
{
get => _account;
set => _account = value;
set
{
_account = value;
// 0x91 credentials just verified: the character list and the world-entry burst follow.
// The login-server pass stays on the initial buffers; it never sends more than a server list.
// An off-loop password check can land its verdict after HandleReceive already flipped the
// state to LoggedIn, so both states promote; both calls below are no-ops once applied.
if (value != null && _protocolState is ProtocolState.GameServer_AwaitingGameServerLogin or ProtocolState.GameServer_LoggedIn)
{
PromoteBuffers();
}
}
}
private void PromoteBuffers()
{
if (_socket == null)
{
return;
}
if (!_socketManager.TryPromoteSendBuffer(_socket) && _socket.SendBuffer.PhysicalSize < SendBufferSize)
{
// The send path promotes on demand and disconnects if that fails too
logger.Debug("{NetState}: send buffer promotion deferred to the send path", this);
}
if (!_socketManager.TryPromoteRecvBuffer(_socket) && _socket.RecvBuffer.PhysicalSize < RecvBufferSize)
{
// The oversize-packet guard in HandlePacket gets one more try where the small buffer matters
logger.Debug("{NetState}: recv buffer promotion deferred", this);
}
}
public string Assistant { get; set; }
@ -525,6 +557,19 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return false;
}
if (_socket.SendBuffer.PhysicalSize < SendBufferSize)
{
// Promotion is unbudgeted and is not growth, so neither the ceiling nor the shrink
// bookkeeping applies. Before credentials verify nothing promotes: the 4 KiB ring is the
// whole pre-auth send budget, and a connection that exceeds it is dropped as exhausted.
if (_account == null)
{
return false;
}
return _socketManager.TryPromoteSendBuffer(_socket);
}
if (!UnderMemoryCeiling())
{
_ceilingRefusals++;
@ -1084,6 +1129,13 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
break;
}
}
// A completion that fills the buffer arms no receive; whatever the loop consumed is free
// space again. No-op while a receive is armed or the buffer is still full.
if (_running)
{
_socket.ResumeReceive();
}
}
catch (Exception ex)
{
@ -1137,6 +1189,22 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
}
}
// Can never complete: the buffer holds PhysicalSize - 1 bytes (one slot tells full from
// empty) and a recv arms only into free space
if (packetLength >= _socket.RecvBuffer.PhysicalSize)
{
// A verified account whose promotion could not be applied earlier gets one more try here,
// where the small buffer actually matters; the swap lands before the next completion's event
if (_account != null && _socket.RecvBuffer.PhysicalSize < RecvBufferSize &&
_socketManager.TryPromoteRecvBuffer(_socket))
{
return ParserState.AwaitingPartialPacket;
}
LogInfo($"Received packet 0x{packetId:X2} declaring {packetLength} bytes, more than the receive buffer holds.");
return ParserState.Error;
}
// Not enough data, let's wait for more to come in
if (length < packetLength)
{

View file

@ -34,7 +34,7 @@
</Target>
<ItemGroup>
<ProjectReference Include="..\Logger\Logger.csproj" />
<PackageReference Include="IORingGroup" Version="1.0.12" />
<PackageReference Include="IORingGroup" Version="1.0.13" />
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
<PackageReference Include="LibDeflate.Bindings" Version="1.0.4" />
<PackageReference Include="System.IO.Hashing" Version="10.0.12" />

View file

@ -68,13 +68,34 @@ Optional systems can add substantially more. The pathfinding prebake
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
small hosts.
Network buffers are 64 KB receive plus a configurable send buffer per connection. Send memory is
`network.sendBufferSize` at rest and can grow to `network.sendBufferMaxSize` under load. Shared
send-buffer tier memory is capped by `network.sendBufferGrowthBudget`, and growth is refused when
process memory exceeds `network.memoryCeilingPercent` of available memory. The worst case is the
base send-buffer size times the connection count, plus the shared growth budget: at the defaults,
100 players is roughly 32 MB at rest, and the growth budget can add up to another 256 MB under
load.
Network buffers come from four pools that grow and shrink with the population rather than being
sized for a full shard. A connection that has not yet presented valid credentials holds a 4 KB
receive and a 4 KB send buffer (the platform's page size). On Windows Server 2012 R2 / 2016 the
transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is
64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. Everything the server
sends before that must fit in that ring — a connection that overruns it is dropped; the stock login
sequence uses under 2 KB. Otherwise, when the game server verifies the account the connection is
promoted to a 64 KB receive buffer and a `network.sendBufferSize` send buffer from the base pools,
and nothing ever moves back. A flood of unauthenticated connections tops out at about 32 MB across
the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and
the base pools used by logged-in players are separate), and never allocates a base-pool slab.
At boot the network holds `network.initialBufferSlabs` slab(s) of each pool — at the defaults one
2 MB receive slab, one 8 MB send slab and two 128 KB pre-auth slabs, about 10 MB — and allocates
another slab only when the population needs one. Each slab covers 32 connections at the
4096-connection maximum. After 15 quiet minutes idle slabs are trimmed back towards current usage,
never past the last 15 minutes' peak, at one slab per pool per minute and never below
`network.initialBufferSlabs`. Only the newest slab is trimmed, and buffers are handed out from the
oldest slab first, so ordinary churn empties the newest slabs; a shard that drops from 4096 players
to a handful takes about two hours to shrink fully, longer if a long-lived connection still holds a
buffer in a newer slab.
Send memory per authenticated connection is `network.sendBufferSize` at rest and can grow to
`network.sendBufferMaxSize` under load. Shared send-buffer tier memory is capped by
`network.sendBufferGrowthBudget`, and growth is refused when process memory exceeds
`network.memoryCeilingPercent` of available memory. The worst case is the receive and base
send-buffer sizes times the number of logged-in connections, plus the shared growth budget: a full
4096 logged-in connections is roughly 1.25 GB of base buffers, and the growth budget can add up to
another 256 MB.
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
Server GC on a 2-core box.
@ -118,6 +139,8 @@ See the README for the full supported list. Two things are worth calling out:
| `network.sendBufferMaxSize` | 2 MB (`2097152`) | Ceiling a single connection's send buffer can grow to under load. Lower it on memory-constrained hosts; raise it if slow clients are disconnected with "send buffer exhausted". |
| `network.sendBufferGrowthBudget` | 256 MB (`268435456`) | Cap on the shared memory the larger send-buffer tiers may use. Lower it on memory-constrained hosts. |
| `network.memoryCeilingPercent` | 80% | Refuse send-buffer growth once the process is above this share of available memory; 0 turns the check off. |
| `network.initialBufferSlabs` | `1` | Slabs of each base pool held from boot, and the floor the trim never goes below. Raise it on a large shard to pre-warm the pools instead of paying for a slab as the population climbs. |
| `network.maxBufferSlabs` | `128` | Divides the connection maximum into base-pool slabs: a slab holds `MaxConnections / maxBufferSlabs` connections, 32 at the default. Raise it for finer slabs on a small host (the slab floor is 16 buffers); lowering it makes each slab, and the boot allocation, larger. It is not a connection or memory cap — both pools still reach the connection maximum. |
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
## Am I undersized?