feat(network): lean base pools (#2641)
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run
Some checks are pending
Build / Build (MacOS 15) (push) Waiting to run
Build / Build (MacOS 26) (push) Waiting to run
Build / Build (AlmaLinux 10) (push) Waiting to run
Build / Build (Debian 12) (push) Waiting to run
Build / Build (Debian 13) (push) Waiting to run
Build / Build (Fedora 44) (push) Waiting to run
Build / Build (CentOS 10 Stream) (push) Waiting to run
Build / Build (CentOS 9 Stream) (push) Waiting to run
Build / Build (Ubuntu 26) (push) Waiting to run
Build / Build (Ubuntu 22) (push) Waiting to run
Build / Build (Ubuntu 24) (push) Waiting to run
**Follow-on to #2639 (merged). References a local IORingGroup `1.0.13-preview.11` pack until 1.0.13 (modernuo/IORingGroup#15) is published; do not merge before that switch.** ## Summary Consumes IORingGroup's lean base pools (modernuo/IORingGroup#15): both network pools now start with one slab, grow a slab at a time with the population, and trim idle slabs back after quiet periods. - Fixes the transport's send-pool cap: previously only 1024 of the 4096 connections could get a send buffer; connection 1025 was closed at accept. - Network memory at boot drops from about 96 MB to about 10 MB at the defaults; a full 4096 logged-in connections is about 1.25 GB of base buffers plus the growth budget. - New settings: `network.initialBufferSlabs` (default 1; slabs of each pool held from boot and the trim floor) and `network.maxBufferSlabs` (default 128; divides the connection maximum into slabs, 32 connections per slab). Both are coerced with a warning; the same value feeds the ring table and the manager so they cannot drift. - The Debug-only maintenance line includes base-pool capacity and releases. - `dev-docs/server-requirements.md` rewrites the network memory story and adds the two settings. ## Pre-auth buffers Every connection starts on the transport's platform-minimum buffers (4 KB receive, 4 KB send) instead of the base pools. It is promoted to full-size buffers (64 KB receive, `network.sendBufferSize` send) when the game server verifies its account — the point where `NetState.Account` is assigned in the `GameServer_AwaitingGameServerLogin` or `GameServer_LoggedIn` state, so a verdict that lands after the parser has moved on still promotes. Nothing ever moves back. The login-server pass stays on the small buffers for its whole lifetime. Before credentials verify, nothing promotes: the 4 KB send ring is the entire pre-auth send budget, and a connection that overruns it is dropped as exhausted, exactly as the receive side drops a packet header declaring more bytes than the receive buffer can hold (new guard in `HandlePacket`; it also closes the old 65535-byte edge on 64 KB buffers). The stock login sequence sends under 2 KB. After credentials verify, the send path promotes on demand if it ever needs to (unbudgeted, outside the memory ceiling and the shrink bookkeeping — promotion is not growth), and the oversize-packet guard waits on a pending receive promotion or retries a stalled one once for a verified account before disconnecting. A completion that fills the receive buffer arms no receive, so `HandleReceive` now calls `RingSocket.ResumeReceive()` after the parse loop — at 4 KB a burst of small packets fills the buffer in one completion. Net effect: a flood of unauthenticated connections tops out at about 32 MB across the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and the base pools used by logged-in players are separate), and never allocates a base-pool slab. Platform note: on Windows Server 2012 R2 / 2016 the transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is 64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. The server logs the effective sizes at startup. ## Testing Server.Tests (905) and UOContent.Tests green on the preview pack (one pre-existing `FamiliarAITests` failure from #2644 reproduces on `main`, tracked separately). New tests cover both coercions, that the ring's registration table equals `RequiredRegisteredBuffers` for the configured values, promotion on game-server auth and on a late account, no promotion on the login server, pre-credential overrun ending in exhaustion, post-credential on-demand promotion, the oversize-packet guard through loopback (error, wait, retry with an account, and a promotion made pending mid-parse), and receiving again after a burst fills the initial buffer.
This commit is contained in:
parent
9d9e672a09
commit
24bcfee554
6 changed files with 682 additions and 22 deletions
|
|
@ -1,4 +1,5 @@
|
|||
using System;
|
||||
using System.Buffers;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using System.Net.Sockets;
|
||||
|
|
@ -12,9 +13,11 @@ namespace Server.Tests.Network;
|
|||
[Collection("Sequential Server Tests")]
|
||||
public class NetStateSendBufferTests
|
||||
{
|
||||
// Authentication is the 0x91 credentials check on the game server; that is where buffers promote
|
||||
private static NetState CreateAuthenticatedNetState(out Socket client)
|
||||
{
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out client);
|
||||
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
|
||||
ns.Account = new MockAccount();
|
||||
return ns;
|
||||
}
|
||||
|
|
@ -26,6 +29,183 @@ public class NetStateSendBufferTests
|
|||
return data;
|
||||
}
|
||||
|
||||
private static unsafe void RegisterNoOpPing()
|
||||
{
|
||||
if (IncomingPackets.GetHandler(0x73) == null)
|
||||
{
|
||||
IncomingPackets.Register(0x73, 2, false, &NoOp);
|
||||
}
|
||||
}
|
||||
|
||||
private static void NoOp(NetState state, SpanReader reader)
|
||||
{
|
||||
}
|
||||
|
||||
private static void SliceUntil(Func<bool> done)
|
||||
{
|
||||
var deadline = Stopwatch.StartNew();
|
||||
while (!done() && deadline.ElapsedMilliseconds < 5000)
|
||||
{
|
||||
NetState.Slice();
|
||||
Thread.Sleep(5);
|
||||
}
|
||||
|
||||
Assert.True(done());
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Account_InTheGameServerState_PromotesBothBuffers()
|
||||
{
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
RegisterNoOpPing();
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
|
||||
|
||||
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
|
||||
ns.Account = new MockAccount();
|
||||
|
||||
// Send promotes at once; nothing was in flight so no buffer retires
|
||||
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
Assert.False(ns._sendBufferGrown);
|
||||
|
||||
// Recv promotes at the next completion
|
||||
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||
client.Send(new byte[] { 0x73, 0x01 });
|
||||
SliceUntil(() => ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize);
|
||||
Assert.True(ns.Running);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Account_OnTheLoginServer_KeepsTheInitialBuffers()
|
||||
{
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
ns._protocolState = NetState.ProtocolState.LoginServer_AwaitingLogin;
|
||||
ns.Account = new MockAccount();
|
||||
|
||||
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Send_BeyondTheInitialBuffer_BeforeCredentials_IsExhausted()
|
||||
{
|
||||
// Nothing promotes before credentials verify: the 4 KiB ring is the whole pre-auth budget
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
|
||||
ns.Send(data);
|
||||
|
||||
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
|
||||
// A graceful server-side close half-closes and waits for the peer's own FIN; closing the
|
||||
// peer here stands in for the client eventually going away, so Running can be observed.
|
||||
// Once it completes the NetState is disposed and _socket goes null, so nothing below
|
||||
// this point may touch it.
|
||||
client.Close();
|
||||
SliceUntil(() => !ns.Running);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Send_BeyondTheInitialBuffer_WithCredentials_PromotesOnDemand()
|
||||
{
|
||||
// The late-verdict path from Account_AssignedAfterTheStateFlipped_StillPromotes: an account
|
||||
// attached while the setter's own gate does not fire must still let the send path promote
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
ns.Account = new MockAccount(); // _protocolState is still AwaitingSeed: the setter does not promote
|
||||
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
|
||||
ns.Send(data);
|
||||
|
||||
Assert.True(ns.Running);
|
||||
Assert.Equal(string.Empty, ns._disconnectReason);
|
||||
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
Assert.False(ns._sendBufferGrown); // promotion is not growth: nothing to shrink later
|
||||
Assert.Equal(data, ReadAll(client, data.Length));
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Send_BacklogBeforeCredentials_IsExhaustedInsteadOfPromoted()
|
||||
{
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
// No Slice() between sends: nothing drains, so a second write finds a non-empty initial buffer
|
||||
var half = Pattern(NetState.InitialSendBufferSize / 2, 11);
|
||||
ns.Send(half);
|
||||
ns.Send(half);
|
||||
ns.Send(half); // cannot fit, buffer not empty, no account: refused
|
||||
|
||||
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
// A graceful server-side close waits for the queued bytes to drain and then for the
|
||||
// peer's own FIN; closing the peer here stands in for the client eventually going away.
|
||||
// Once it completes the NetState is disposed and _socket goes null, so nothing below
|
||||
// this point may touch it.
|
||||
client.Close();
|
||||
SliceUntil(() => !ns.Running);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void Account_AssignedAfterTheStateFlipped_StillPromotes()
|
||||
{
|
||||
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||
try
|
||||
{
|
||||
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||
ns.Account = new MockAccount();
|
||||
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ns.Dispose();
|
||||
client.Close();
|
||||
}
|
||||
}
|
||||
|
||||
// A random prefix lands under the target; zeros (2 bits each) walk it up a byte at a time
|
||||
private static byte[] CompressesToExactly(int compressedLength, int seed)
|
||||
{
|
||||
|
|
@ -448,4 +628,58 @@ public class NetStateSendBufferTests
|
|||
// growth off; budget untouched
|
||||
Assert.Equal(1L, NetState.CoerceSendBufferGrowthBudget(1, sendBufferSize, sendBufferSize));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CoerceMaxBufferSlabs_ClampsToOneSlabPerConnection()
|
||||
{
|
||||
var maxConnections = NetState.SocketManager.MaxSockets;
|
||||
|
||||
// fewer than one slab is meaningless
|
||||
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(0));
|
||||
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(-4));
|
||||
|
||||
// more slabs than connections cannot make a slab any smaller
|
||||
Assert.Equal(maxConnections, NetState.CoerceMaxBufferSlabs(maxConnections * 2));
|
||||
|
||||
Assert.Equal(128, NetState.CoerceMaxBufferSlabs(128));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CoerceInitialBufferSlabs_ClampsToTheSlabCount()
|
||||
{
|
||||
var slabs = NetState.BasePoolSlabCount(128);
|
||||
Assert.True(slabs > 1);
|
||||
|
||||
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(0, slabs));
|
||||
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(-1, slabs));
|
||||
|
||||
// a pool never holds more slabs than it has
|
||||
Assert.Equal(slabs, NetState.CoerceInitialBufferSlabs(slabs + 1, slabs));
|
||||
|
||||
Assert.Equal(4, NetState.CoerceInitialBufferSlabs(4, slabs));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ring_RegisteredBufferTable_MatchesTheConfiguredSlabCount()
|
||||
{
|
||||
var manager = NetState.SocketManager;
|
||||
var maxBufferSlabs = NetState.CoerceMaxBufferSlabs(ServerConfiguration.GetSetting("network.maxBufferSlabs", NetState.DefaultMaxBufferSlabs));
|
||||
|
||||
// A table smaller than this throws at manager construction. Connections start on the
|
||||
// transport minimum until the game server promotes them, so the formula must count those too.
|
||||
// Mirrors what production passes: the request, not the effective size the manager may coerce
|
||||
// down to (RequiredRegisteredBuffers treats a non-zero request as a pool either way).
|
||||
Assert.Equal(
|
||||
RingSocketManager.RequiredRegisteredBuffers(
|
||||
manager.MaxSockets,
|
||||
NetState.SendBufferSize,
|
||||
manager.MaxSendBufferSize,
|
||||
manager.SendBufferGrowthBudget,
|
||||
maxBufferSlabs,
|
||||
IORingBuffer.MinimumSize,
|
||||
IORingBuffer.MinimumSize
|
||||
),
|
||||
NetState.Ring.MaxRegisteredBuffers
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue