feat(network): lean base pools (#2641)
Some checks failed
Build / Build (MacOS 15) (push) Has been cancelled
Build / Build (MacOS 26) (push) Has been cancelled
Build / Build (AlmaLinux 10) (push) Has been cancelled
Build / Build (Debian 12) (push) Has been cancelled
Build / Build (Debian 13) (push) Has been cancelled
Build / Build (Fedora 44) (push) Has been cancelled
Build / Build (CentOS 10 Stream) (push) Has been cancelled
Build / Build (CentOS 9 Stream) (push) Has been cancelled
Build / Build (Ubuntu 26) (push) Has been cancelled
Build / Build (Ubuntu 22) (push) Has been cancelled
Build / Build (Ubuntu 24) (push) Has been cancelled
Some checks failed
Build / Build (MacOS 15) (push) Has been cancelled
Build / Build (MacOS 26) (push) Has been cancelled
Build / Build (AlmaLinux 10) (push) Has been cancelled
Build / Build (Debian 12) (push) Has been cancelled
Build / Build (Debian 13) (push) Has been cancelled
Build / Build (Fedora 44) (push) Has been cancelled
Build / Build (CentOS 10 Stream) (push) Has been cancelled
Build / Build (CentOS 9 Stream) (push) Has been cancelled
Build / Build (Ubuntu 26) (push) Has been cancelled
Build / Build (Ubuntu 22) (push) Has been cancelled
Build / Build (Ubuntu 24) (push) Has been cancelled
**Follow-on to #2639 (merged). References a local IORingGroup `1.0.13-preview.11` pack until 1.0.13 (modernuo/IORingGroup#15) is published; do not merge before that switch.** ## Summary Consumes IORingGroup's lean base pools (modernuo/IORingGroup#15): both network pools now start with one slab, grow a slab at a time with the population, and trim idle slabs back after quiet periods. - Fixes the transport's send-pool cap: previously only 1024 of the 4096 connections could get a send buffer; connection 1025 was closed at accept. - Network memory at boot drops from about 96 MB to about 10 MB at the defaults; a full 4096 logged-in connections is about 1.25 GB of base buffers plus the growth budget. - New settings: `network.initialBufferSlabs` (default 1; slabs of each pool held from boot and the trim floor) and `network.maxBufferSlabs` (default 128; divides the connection maximum into slabs, 32 connections per slab). Both are coerced with a warning; the same value feeds the ring table and the manager so they cannot drift. - The Debug-only maintenance line includes base-pool capacity and releases. - `dev-docs/server-requirements.md` rewrites the network memory story and adds the two settings. ## Pre-auth buffers Every connection starts on the transport's platform-minimum buffers (4 KB receive, 4 KB send) instead of the base pools. It is promoted to full-size buffers (64 KB receive, `network.sendBufferSize` send) when the game server verifies its account — the point where `NetState.Account` is assigned in the `GameServer_AwaitingGameServerLogin` or `GameServer_LoggedIn` state, so a verdict that lands after the parser has moved on still promotes. Nothing ever moves back. The login-server pass stays on the small buffers for its whole lifetime. Before credentials verify, nothing promotes: the 4 KB send ring is the entire pre-auth send budget, and a connection that overruns it is dropped as exhausted, exactly as the receive side drops a packet header declaring more bytes than the receive buffer can hold (new guard in `HandlePacket`; it also closes the old 65535-byte edge on 64 KB buffers). The stock login sequence sends under 2 KB. After credentials verify, the send path promotes on demand if it ever needs to (unbudgeted, outside the memory ceiling and the shrink bookkeeping — promotion is not growth), and the oversize-packet guard waits on a pending receive promotion or retries a stalled one once for a verified account before disconnecting. A completion that fills the receive buffer arms no receive, so `HandleReceive` now calls `RingSocket.ResumeReceive()` after the parse loop — at 4 KB a burst of small packets fills the buffer in one completion. Net effect: a flood of unauthenticated connections tops out at about 32 MB across the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and the base pools used by logged-in players are separate), and never allocates a base-pool slab. Platform note: on Windows Server 2012 R2 / 2016 the transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is 64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. The server logs the effective sizes at startup. ## Testing Server.Tests (905) and UOContent.Tests green on the preview pack (one pre-existing `FamiliarAITests` failure from #2644 reproduces on `main`, tracked separately). New tests cover both coercions, that the ring's registration table equals `RequiredRegisteredBuffers` for the configured values, promotion on game-server auth and on a late account, no promotion on the login server, pre-credential overrun ending in exhaustion, post-credential on-demand promotion, the oversize-packet guard through loopback (error, wait, retry with an account, and a promotion made pending mid-parse), and receiving again after a burst fills the initial buffer.
This commit is contained in:
parent
9d9e672a09
commit
24bcfee554
6 changed files with 682 additions and 22 deletions
236
Projects/Server.Tests/Tests/Network/NetStateRecvGuardTests.cs
Normal file
236
Projects/Server.Tests/Tests/Network/NetStateRecvGuardTests.cs
Normal file
|
|
@ -0,0 +1,236 @@
|
||||||
|
using System;
|
||||||
|
using System.Buffers;
|
||||||
|
using System.Diagnostics;
|
||||||
|
using System.Network;
|
||||||
|
using System.Threading;
|
||||||
|
using Server.Network;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace Server.Tests.Network;
|
||||||
|
|
||||||
|
[Collection("Sequential Server Tests")]
|
||||||
|
public class NetStateRecvGuardTests
|
||||||
|
{
|
||||||
|
private const byte TestPacketId = 0xB1;
|
||||||
|
|
||||||
|
private static unsafe void RegisterVariableLength()
|
||||||
|
{
|
||||||
|
if (IncomingPackets.GetHandler(TestPacketId) == null)
|
||||||
|
{
|
||||||
|
IncomingPackets.Register(TestPacketId, 0, false, &NoOp);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void NoOp(NetState state, SpanReader reader)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
// Not 0x73: NetStateSendBufferTests registers its own no-op there first-come-wins, since the
|
||||||
|
// handler table is process-global, and this test needs its own handler's side effect to fire.
|
||||||
|
private const byte TestPingPacketId = 0x72;
|
||||||
|
|
||||||
|
private static byte _lastPing;
|
||||||
|
|
||||||
|
private static unsafe void RegisterNoOpPing()
|
||||||
|
{
|
||||||
|
if (IncomingPackets.GetHandler(TestPingPacketId) == null)
|
||||||
|
{
|
||||||
|
IncomingPackets.Register(TestPingPacketId, 2, false, &RecordPing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void RecordPing(NetState state, SpanReader reader) => _lastPing = reader.ReadByte();
|
||||||
|
|
||||||
|
private const byte AttachAccountPacketId = 0x71;
|
||||||
|
|
||||||
|
private static unsafe void RegisterAttachAccount()
|
||||||
|
{
|
||||||
|
if (IncomingPackets.GetHandler(AttachAccountPacketId) == null)
|
||||||
|
{
|
||||||
|
IncomingPackets.Register(AttachAccountPacketId, 3, false, &AttachAccount);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stands in for the 0x91 handler: the account attaches mid-parse, so the promotion is pending
|
||||||
|
// with a receive armed when the next packet in the same completion reaches the guard
|
||||||
|
private static void AttachAccount(NetState state, SpanReader reader) => state.Account = new MockAccount();
|
||||||
|
|
||||||
|
private static void SliceUntil(Func<bool> done)
|
||||||
|
{
|
||||||
|
var deadline = Stopwatch.StartNew();
|
||||||
|
while (!done() && deadline.ElapsedMilliseconds < 5000)
|
||||||
|
{
|
||||||
|
NetState.Slice();
|
||||||
|
Thread.Sleep(5);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.True(done());
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void HandleReceive_PacketLongerThanTheRecvBuffer_IsAnError()
|
||||||
|
{
|
||||||
|
RegisterVariableLength();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
// A 16-bit length cannot exceed a 64 KiB buffer, so the guard is unreachable there and
|
||||||
|
// this only ever hit the `< 3` check on a buffer that size.
|
||||||
|
Skip.If(ns._socket.RecvBuffer.PhysicalSize > ushort.MaxValue);
|
||||||
|
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
|
||||||
|
// A variable-length header claiming more than the buffer can ever hold would otherwise
|
||||||
|
// park the connection with nothing to arm a recv into
|
||||||
|
var declared = ns._socket.RecvBuffer.PhysicalSize; // one more than the buffer can ever hold
|
||||||
|
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
|
||||||
|
SliceUntil(() => ns._protocolState == NetState.ProtocolState.Error);
|
||||||
|
|
||||||
|
Assert.Contains("bad state", ns._disconnectReason);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HandleReceive_PacketThatFits_WaitsForTheRest()
|
||||||
|
{
|
||||||
|
RegisterVariableLength();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
|
||||||
|
var declared = ns._socket.RecvBuffer.PhysicalSize / 2;
|
||||||
|
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
|
||||||
|
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 3);
|
||||||
|
|
||||||
|
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
Assert.Equal(3, ns._socket.RecvBuffer.ReadableBytes);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void HandleReceive_OversizePacket_WithAnAccount_RetriesPromotionInsteadOfErroring()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialRecvBufferSize == 0);
|
||||||
|
RegisterVariableLength();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
// _protocolState is still AwaitingSeed here, so the Account setter's gate does not fire
|
||||||
|
ns.Account = new MockAccount();
|
||||||
|
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
|
||||||
|
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
|
||||||
|
var declared = NetState.InitialRecvBufferSize; // as much as the initial buffer can ever hold
|
||||||
|
client.Send(new byte[] { TestPacketId, (byte)(declared >> 8), (byte)declared });
|
||||||
|
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 3);
|
||||||
|
|
||||||
|
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
|
||||||
|
// The deferred promotion applies at this next completion; the header is then delivered
|
||||||
|
// whole to the 0xB1 no-op handler
|
||||||
|
client.Send(new byte[declared - 3]);
|
||||||
|
SliceUntil(
|
||||||
|
() => ns._socket.RecvBuffer.ReadableBytes == 0 &&
|
||||||
|
ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void HandleReceive_AfterABurstFillsTheInitialBuffer_KeepsReceiving()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialRecvBufferSize == 0);
|
||||||
|
RegisterNoOpPing();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
|
||||||
|
// One segment of pings fills the 4 KiB buffer (capacity is PhysicalSize - 1, odd, so the
|
||||||
|
// last ping straddles two completions); the parser consumes them all
|
||||||
|
var capacity = ns._socket.RecvBuffer.PhysicalSize - 1;
|
||||||
|
var pings = new byte[capacity + 1];
|
||||||
|
for (var i = 0; i < pings.Length; i += 2)
|
||||||
|
{
|
||||||
|
pings[i] = TestPingPacketId;
|
||||||
|
pings[i + 1] = (byte)(i / 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
client.Send(pings);
|
||||||
|
SliceUntil(() => ns._socket.RecvBuffer.ReadableBytes == 0 && ns._receivedData);
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
|
||||||
|
// Without a re-arm this ping never arrives
|
||||||
|
client.Send(new byte[] { TestPingPacketId, 0xEE });
|
||||||
|
SliceUntil(() => _lastPing == 0xEE);
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void HandleReceive_AccountAttachedMidParse_ThenOversizeHeader_WaitsForThePendingPromotion()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialRecvBufferSize == 0);
|
||||||
|
RegisterAttachAccount();
|
||||||
|
RegisterVariableLength();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
var declared = ns._socket.RecvBuffer.PhysicalSize;
|
||||||
|
|
||||||
|
// One send, one completion on loopback: the account packet promotes (deferred: a receive is
|
||||||
|
// armed), then the oversize header reaches the guard with that promotion pending. If the OS
|
||||||
|
// ever splits this into two completions instead, the first assertion block still holds — the
|
||||||
|
// header then hits the guard after the promotion already applied and simply waits on the
|
||||||
|
// 64 KiB buffer, so the test cannot flake, it just exercises the weaker path on that run.
|
||||||
|
client.Send(new byte[] { AttachAccountPacketId, 0x00, 0x00, TestPacketId, (byte)(declared >> 8), (byte)declared });
|
||||||
|
SliceUntil(() => ns.Account != null);
|
||||||
|
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
Assert.Equal(NetState.ProtocolState.GameServer_LoggedIn, ns._protocolState);
|
||||||
|
Assert.Equal(3, ns._socket.RecvBuffer.ReadableBytes); // the header waits, not rejected
|
||||||
|
|
||||||
|
// The next completion applies the promotion and the whole packet is delivered
|
||||||
|
client.Send(new byte[declared - 3]);
|
||||||
|
SliceUntil(
|
||||||
|
() => ns._socket.RecvBuffer.ReadableBytes == 0 &&
|
||||||
|
ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
using System;
|
using System;
|
||||||
|
using System.Buffers;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Diagnostics;
|
using System.Diagnostics;
|
||||||
using System.Net.Sockets;
|
using System.Net.Sockets;
|
||||||
|
|
@ -12,9 +13,11 @@ namespace Server.Tests.Network;
|
||||||
[Collection("Sequential Server Tests")]
|
[Collection("Sequential Server Tests")]
|
||||||
public class NetStateSendBufferTests
|
public class NetStateSendBufferTests
|
||||||
{
|
{
|
||||||
|
// Authentication is the 0x91 credentials check on the game server; that is where buffers promote
|
||||||
private static NetState CreateAuthenticatedNetState(out Socket client)
|
private static NetState CreateAuthenticatedNetState(out Socket client)
|
||||||
{
|
{
|
||||||
var ns = PacketTestUtilities.CreateTestNetState(out client);
|
var ns = PacketTestUtilities.CreateTestNetState(out client);
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
|
||||||
ns.Account = new MockAccount();
|
ns.Account = new MockAccount();
|
||||||
return ns;
|
return ns;
|
||||||
}
|
}
|
||||||
|
|
@ -26,6 +29,183 @@ public class NetStateSendBufferTests
|
||||||
return data;
|
return data;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static unsafe void RegisterNoOpPing()
|
||||||
|
{
|
||||||
|
if (IncomingPackets.GetHandler(0x73) == null)
|
||||||
|
{
|
||||||
|
IncomingPackets.Register(0x73, 2, false, &NoOp);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void NoOp(NetState state, SpanReader reader)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void SliceUntil(Func<bool> done)
|
||||||
|
{
|
||||||
|
var deadline = Stopwatch.StartNew();
|
||||||
|
while (!done() && deadline.ElapsedMilliseconds < 5000)
|
||||||
|
{
|
||||||
|
NetState.Slice();
|
||||||
|
Thread.Sleep(5);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.True(done());
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Account_InTheGameServerState_PromotesBothBuffers()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
RegisterNoOpPing();
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
|
||||||
|
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_AwaitingGameServerLogin;
|
||||||
|
ns.Account = new MockAccount();
|
||||||
|
|
||||||
|
// Send promotes at once; nothing was in flight so no buffer retires
|
||||||
|
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
Assert.False(ns._sendBufferGrown);
|
||||||
|
|
||||||
|
// Recv promotes at the next completion
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
client.Send(new byte[] { 0x73, 0x01 });
|
||||||
|
SliceUntil(() => ns._socket.RecvBuffer.PhysicalSize == NetState.RecvBufferSize);
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Account_OnTheLoginServer_KeepsTheInitialBuffers()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns._protocolState = NetState.ProtocolState.LoginServer_AwaitingLogin;
|
||||||
|
ns.Account = new MockAccount();
|
||||||
|
|
||||||
|
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
Assert.Equal(NetState.InitialRecvBufferSize, ns._socket.RecvBuffer.PhysicalSize);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Send_BeyondTheInitialBuffer_BeforeCredentials_IsExhausted()
|
||||||
|
{
|
||||||
|
// Nothing promotes before credentials verify: the 4 KiB ring is the whole pre-auth budget
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
|
||||||
|
ns.Send(data);
|
||||||
|
|
||||||
|
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
|
||||||
|
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
|
||||||
|
// A graceful server-side close half-closes and waits for the peer's own FIN; closing the
|
||||||
|
// peer here stands in for the client eventually going away, so Running can be observed.
|
||||||
|
// Once it completes the NetState is disposed and _socket goes null, so nothing below
|
||||||
|
// this point may touch it.
|
||||||
|
client.Close();
|
||||||
|
SliceUntil(() => !ns.Running);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Send_BeyondTheInitialBuffer_WithCredentials_PromotesOnDemand()
|
||||||
|
{
|
||||||
|
// The late-verdict path from Account_AssignedAfterTheStateFlipped_StillPromotes: an account
|
||||||
|
// attached while the setter's own gate does not fire must still let the send path promote
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns.Account = new MockAccount(); // _protocolState is still AwaitingSeed: the setter does not promote
|
||||||
|
var data = Pattern(NetState.InitialSendBufferSize + 512, 7);
|
||||||
|
ns.Send(data);
|
||||||
|
|
||||||
|
Assert.True(ns.Running);
|
||||||
|
Assert.Equal(string.Empty, ns._disconnectReason);
|
||||||
|
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
Assert.False(ns._sendBufferGrown); // promotion is not growth: nothing to shrink later
|
||||||
|
Assert.Equal(data, ReadAll(client, data.Length));
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Send_BacklogBeforeCredentials_IsExhaustedInsteadOfPromoted()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
// No Slice() between sends: nothing drains, so a second write finds a non-empty initial buffer
|
||||||
|
var half = Pattern(NetState.InitialSendBufferSize / 2, 11);
|
||||||
|
ns.Send(half);
|
||||||
|
ns.Send(half);
|
||||||
|
ns.Send(half); // cannot fit, buffer not empty, no account: refused
|
||||||
|
|
||||||
|
Assert.Equal(NetState.InitialSendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
Assert.Contains("exhausted", ns._disconnectReason, StringComparison.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
// A graceful server-side close waits for the queued bytes to drain and then for the
|
||||||
|
// peer's own FIN; closing the peer here stands in for the client eventually going away.
|
||||||
|
// Once it completes the NetState is disposed and _socket goes null, so nothing below
|
||||||
|
// this point may touch it.
|
||||||
|
client.Close();
|
||||||
|
SliceUntil(() => !ns.Running);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public void Account_AssignedAfterTheStateFlipped_StillPromotes()
|
||||||
|
{
|
||||||
|
Skip.If(NetState.InitialSendBufferSize == 0);
|
||||||
|
var ns = PacketTestUtilities.CreateTestNetState(out var client);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
ns._protocolState = NetState.ProtocolState.GameServer_LoggedIn;
|
||||||
|
ns.Account = new MockAccount();
|
||||||
|
Assert.Equal(NetState.SendBufferSize, ns._socket.SendBuffer.PhysicalSize);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ns.Dispose();
|
||||||
|
client.Close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A random prefix lands under the target; zeros (2 bits each) walk it up a byte at a time
|
// A random prefix lands under the target; zeros (2 bits each) walk it up a byte at a time
|
||||||
private static byte[] CompressesToExactly(int compressedLength, int seed)
|
private static byte[] CompressesToExactly(int compressedLength, int seed)
|
||||||
{
|
{
|
||||||
|
|
@ -448,4 +628,58 @@ public class NetStateSendBufferTests
|
||||||
// growth off; budget untouched
|
// growth off; budget untouched
|
||||||
Assert.Equal(1L, NetState.CoerceSendBufferGrowthBudget(1, sendBufferSize, sendBufferSize));
|
Assert.Equal(1L, NetState.CoerceSendBufferGrowthBudget(1, sendBufferSize, sendBufferSize));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void CoerceMaxBufferSlabs_ClampsToOneSlabPerConnection()
|
||||||
|
{
|
||||||
|
var maxConnections = NetState.SocketManager.MaxSockets;
|
||||||
|
|
||||||
|
// fewer than one slab is meaningless
|
||||||
|
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(0));
|
||||||
|
Assert.Equal(1, NetState.CoerceMaxBufferSlabs(-4));
|
||||||
|
|
||||||
|
// more slabs than connections cannot make a slab any smaller
|
||||||
|
Assert.Equal(maxConnections, NetState.CoerceMaxBufferSlabs(maxConnections * 2));
|
||||||
|
|
||||||
|
Assert.Equal(128, NetState.CoerceMaxBufferSlabs(128));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void CoerceInitialBufferSlabs_ClampsToTheSlabCount()
|
||||||
|
{
|
||||||
|
var slabs = NetState.BasePoolSlabCount(128);
|
||||||
|
Assert.True(slabs > 1);
|
||||||
|
|
||||||
|
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(0, slabs));
|
||||||
|
Assert.Equal(1, NetState.CoerceInitialBufferSlabs(-1, slabs));
|
||||||
|
|
||||||
|
// a pool never holds more slabs than it has
|
||||||
|
Assert.Equal(slabs, NetState.CoerceInitialBufferSlabs(slabs + 1, slabs));
|
||||||
|
|
||||||
|
Assert.Equal(4, NetState.CoerceInitialBufferSlabs(4, slabs));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Ring_RegisteredBufferTable_MatchesTheConfiguredSlabCount()
|
||||||
|
{
|
||||||
|
var manager = NetState.SocketManager;
|
||||||
|
var maxBufferSlabs = NetState.CoerceMaxBufferSlabs(ServerConfiguration.GetSetting("network.maxBufferSlabs", NetState.DefaultMaxBufferSlabs));
|
||||||
|
|
||||||
|
// A table smaller than this throws at manager construction. Connections start on the
|
||||||
|
// transport minimum until the game server promotes them, so the formula must count those too.
|
||||||
|
// Mirrors what production passes: the request, not the effective size the manager may coerce
|
||||||
|
// down to (RequiredRegisteredBuffers treats a non-zero request as a pool either way).
|
||||||
|
Assert.Equal(
|
||||||
|
RingSocketManager.RequiredRegisteredBuffers(
|
||||||
|
manager.MaxSockets,
|
||||||
|
NetState.SendBufferSize,
|
||||||
|
manager.MaxSendBufferSize,
|
||||||
|
manager.SendBufferGrowthBudget,
|
||||||
|
maxBufferSlabs,
|
||||||
|
IORingBuffer.MinimumSize,
|
||||||
|
IORingBuffer.MinimumSize
|
||||||
|
),
|
||||||
|
NetState.Ring.MaxRegisteredBuffers
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -29,18 +29,26 @@ namespace Server.Network;
|
||||||
public partial class NetState
|
public partial class NetState
|
||||||
{
|
{
|
||||||
// Buffer sizes
|
// Buffer sizes
|
||||||
private const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
|
internal const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
|
||||||
private const int DefaultSendBufferSize = 1024 * 256; // 256KB send buffers
|
private const int DefaultSendBufferSize = 1024 * 256; // 256KB send buffers
|
||||||
private const int MinSendBufferSize = 1024 * 64; // Platform allocation granularity
|
private const int MinSendBufferSize = 1024 * 64; // Platform allocation granularity
|
||||||
private const int DefaultMaxSendBufferSize = 1024 * 1024 * 2; // 2 MB
|
private const int DefaultMaxSendBufferSize = 1024 * 1024 * 2; // 2 MB
|
||||||
private const long DefaultSendBufferGrowthBudget = 1024L * 1024 * 256; // 256 MB
|
private const long DefaultSendBufferGrowthBudget = 1024L * 1024 * 256; // 256 MB
|
||||||
private const int DefaultMemoryCeilingPercent = 80;
|
private const int DefaultMemoryCeilingPercent = 80;
|
||||||
|
private const int DefaultInitialBufferSlabs = 1; // one slab of each base pool warm at boot
|
||||||
|
internal const int DefaultMaxBufferSlabs = 128; // 32 connections per slab at MaxConnections
|
||||||
|
|
||||||
// Transport ceiling; larger values overflow its tier enumeration
|
// Transport ceiling; larger values overflow its tier enumeration
|
||||||
private const int TransportMaxSendBufferSize = 1024 * 1024 * 256; // 256 MB
|
private const int TransportMaxSendBufferSize = 1024 * 1024 * 256; // 256 MB
|
||||||
private const int MaxConnections = 4096; // Max concurrent connections
|
private const int MaxConnections = 4096; // Max concurrent connections
|
||||||
|
|
||||||
|
internal static int SendBufferSize { get; private set; }
|
||||||
internal static int MaxSendBufferSize { get; private set; }
|
internal static int MaxSendBufferSize { get; private set; }
|
||||||
|
|
||||||
|
// Pre-auth buffers are the smallest the platform can map; a platform whose floor is not below
|
||||||
|
// the base size (the Windows legacy mapping path) starts sockets on base
|
||||||
|
internal static int InitialRecvBufferSize { get; private set; }
|
||||||
|
internal static int InitialSendBufferSize { get; private set; }
|
||||||
private static long _sendBufferGrowthBudget;
|
private static long _sendBufferGrowthBudget;
|
||||||
private static int _memoryCeilingPercent;
|
private static int _memoryCeilingPercent;
|
||||||
|
|
||||||
|
|
@ -51,6 +59,7 @@ public partial class NetState
|
||||||
private static long _lastTierCapacityBytes;
|
private static long _lastTierCapacityBytes;
|
||||||
private static int _lastTierInUse;
|
private static int _lastTierInUse;
|
||||||
private static int _lastTierRetainFloor;
|
private static int _lastTierRetainFloor;
|
||||||
|
private static long _lastBaseCapacityBytes;
|
||||||
|
|
||||||
private static readonly Queue<NetState> _disposed = [];
|
private static readonly Queue<NetState> _disposed = [];
|
||||||
private static readonly TimeSpan ConnectingSocketIdleLimit = TimeSpan.FromMilliseconds(5000); // 5 seconds
|
private static readonly TimeSpan ConnectingSocketIdleLimit = TimeSpan.FromMilliseconds(5000); // 5 seconds
|
||||||
|
|
@ -141,23 +150,46 @@ public partial class NetState
|
||||||
|
|
||||||
// Per-connection send buffer: the lever for "send buffer exhausted" disconnects, and the
|
// Per-connection send buffer: the lever for "send buffer exhausted" disconnects, and the
|
||||||
// per-connection memory ceiling.
|
// per-connection memory ceiling.
|
||||||
var sendBufferSize = GetSendBufferSize();
|
SendBufferSize = GetSendBufferSize();
|
||||||
MaxSendBufferSize = GetPowerOfTwoSetting("network.sendBufferMaxSize", DefaultMaxSendBufferSize, sendBufferSize);
|
MaxSendBufferSize = GetPowerOfTwoSetting("network.sendBufferMaxSize", DefaultMaxSendBufferSize, SendBufferSize);
|
||||||
_sendBufferGrowthBudget = CoerceSendBufferGrowthBudget(
|
_sendBufferGrowthBudget = CoerceSendBufferGrowthBudget(
|
||||||
ServerConfiguration.GetOrUpdateSetting("network.sendBufferGrowthBudget", DefaultSendBufferGrowthBudget),
|
ServerConfiguration.GetOrUpdateSetting("network.sendBufferGrowthBudget", DefaultSendBufferGrowthBudget),
|
||||||
sendBufferSize,
|
SendBufferSize,
|
||||||
MaxSendBufferSize
|
MaxSendBufferSize
|
||||||
);
|
);
|
||||||
// 0 disables the ceiling
|
// 0 disables the ceiling
|
||||||
_memoryCeilingPercent = Math.Clamp(ServerConfiguration.GetOrUpdateSetting("network.memoryCeilingPercent", DefaultMemoryCeilingPercent), 0, 100);
|
_memoryCeilingPercent = Math.Clamp(ServerConfiguration.GetOrUpdateSetting("network.memoryCeilingPercent", DefaultMemoryCeilingPercent), 0, 100);
|
||||||
_availableMemoryBytes = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes;
|
_availableMemoryBytes = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes;
|
||||||
|
|
||||||
const int maxBufferSlabs = 32;
|
// Divides MaxConnections into base-pool slabs; both pools still reach MaxConnections, so
|
||||||
|
// this sets slab granularity, not a connection or memory ceiling.
|
||||||
|
var maxBufferSlabs = CoerceMaxBufferSlabs(
|
||||||
|
ServerConfiguration.GetOrUpdateSetting("network.maxBufferSlabs", DefaultMaxBufferSlabs)
|
||||||
|
);
|
||||||
|
|
||||||
|
// Slabs of each base pool held from boot; the pools grow and trim from here
|
||||||
|
var initialBufferSlabs = CoerceInitialBufferSlabs(
|
||||||
|
ServerConfiguration.GetOrUpdateSetting("network.initialBufferSlabs", DefaultInitialBufferSlabs),
|
||||||
|
BasePoolSlabCount(maxBufferSlabs)
|
||||||
|
);
|
||||||
|
|
||||||
|
// Until the game server verifies credentials a connection holds the smallest buffers the
|
||||||
|
// platform can map; a flood can fill these pools but never reaches the base pools. This is a
|
||||||
|
// request: the manager raises it to the platform floor and turns the pool off if that leaves
|
||||||
|
// no room below the base size (the Windows legacy mapping path floors at 64 KiB, which can
|
||||||
|
// equal RecvBufferSize). The effective sizes are read back below once the manager knows them.
|
||||||
|
var initialBufferSize = IORingBuffer.MinimumSize;
|
||||||
|
|
||||||
|
// Both calls take the same slab count; the manager throws if the table is smaller. Sized by
|
||||||
|
// the request, not the effective size the manager may coerce down to - conservative, never small.
|
||||||
var ring = IORingGroup.Create(
|
var ring = IORingGroup.Create(
|
||||||
queueSize: MaxConnections * 2,
|
queueSize: MaxConnections * 2,
|
||||||
maxConnections: MaxConnections,
|
maxConnections: MaxConnections,
|
||||||
maxOutstandingSends: maxOutstandingSends,
|
maxOutstandingSends: maxOutstandingSends,
|
||||||
maxRegisteredBuffers: RingSocketManager.RequiredRegisteredBuffers(MaxConnections, sendBufferSize, MaxSendBufferSize, _sendBufferGrowthBudget, maxBufferSlabs)
|
maxRegisteredBuffers: RingSocketManager.RequiredRegisteredBuffers(
|
||||||
|
MaxConnections, SendBufferSize, MaxSendBufferSize, _sendBufferGrowthBudget, maxBufferSlabs,
|
||||||
|
initialBufferSize, initialBufferSize
|
||||||
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
// Create socket manager which handles buffer pools and socket lifecycle
|
// Create socket manager which handles buffer pools and socket lifecycle
|
||||||
|
|
@ -165,13 +197,28 @@ public partial class NetState
|
||||||
ring,
|
ring,
|
||||||
maxSockets: MaxConnections,
|
maxSockets: MaxConnections,
|
||||||
recvBufferSize: RecvBufferSize,
|
recvBufferSize: RecvBufferSize,
|
||||||
sendBufferSize: sendBufferSize,
|
sendBufferSize: SendBufferSize,
|
||||||
initialBufferSlabs: 8,
|
initialBufferSlabs: initialBufferSlabs,
|
||||||
maxBufferSlabs: maxBufferSlabs,
|
maxBufferSlabs: maxBufferSlabs,
|
||||||
maxSendBufferSize: MaxSendBufferSize,
|
maxSendBufferSize: MaxSendBufferSize,
|
||||||
sendBufferGrowthBudget: _sendBufferGrowthBudget
|
sendBufferGrowthBudget: _sendBufferGrowthBudget,
|
||||||
|
initialRecvBufferSize: initialBufferSize,
|
||||||
|
initialSendBufferSize: initialBufferSize
|
||||||
);
|
);
|
||||||
|
|
||||||
|
InitialRecvBufferSize = _socketManager.InitialRecvBufferSize;
|
||||||
|
InitialSendBufferSize = _socketManager.InitialSendBufferSize;
|
||||||
|
|
||||||
|
if (InitialRecvBufferSize == 0 || InitialSendBufferSize == 0)
|
||||||
|
{
|
||||||
|
logger.Information(
|
||||||
|
"Pre-auth buffers off where the platform floor ({Minimum} bytes) leaves no room below the base size (recv {Recv}, send {Send})",
|
||||||
|
IORingBuffer.MinimumSize,
|
||||||
|
RecvBufferSize,
|
||||||
|
SendBufferSize
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
_maintenanceTimer = Timer.DelayCall(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1), MaintainSendBuffers);
|
_maintenanceTimer = Timer.DelayCall(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1), MaintainSendBuffers);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -193,27 +240,79 @@ public partial class NetState
|
||||||
var stats = _socketManager.Maintain();
|
var stats = _socketManager.Maintain();
|
||||||
var changed = stats.TierCapacityBytes != _lastTierCapacityBytes ||
|
var changed = stats.TierCapacityBytes != _lastTierCapacityBytes ||
|
||||||
stats.TierInUse != _lastTierInUse ||
|
stats.TierInUse != _lastTierInUse ||
|
||||||
stats.TierRetainFloor != _lastTierRetainFloor;
|
stats.TierRetainFloor != _lastTierRetainFloor ||
|
||||||
|
stats.BaseCapacityBytes != _lastBaseCapacityBytes;
|
||||||
_lastTierCapacityBytes = stats.TierCapacityBytes;
|
_lastTierCapacityBytes = stats.TierCapacityBytes;
|
||||||
_lastTierInUse = stats.TierInUse;
|
_lastTierInUse = stats.TierInUse;
|
||||||
_lastTierRetainFloor = stats.TierRetainFloor;
|
_lastTierRetainFloor = stats.TierRetainFloor;
|
||||||
|
_lastBaseCapacityBytes = stats.BaseCapacityBytes;
|
||||||
|
|
||||||
// Quiet unless something moved
|
// Quiet unless something moved
|
||||||
if (changed || stats.BuffersReleased > 0 || stats.GrowthRefusals > 0 || capRefusals > 0 || ceilingRefusals > 0)
|
if (changed || stats.BuffersReleased > 0 || stats.BaseBuffersReleased > 0 || stats.GrowthRefusals > 0 ||
|
||||||
|
capRefusals > 0 || ceilingRefusals > 0)
|
||||||
{
|
{
|
||||||
logger.Debug(
|
logger.Debug(
|
||||||
"Send buffer tiers: {Capacity} bytes of tier capacity, {InUse} buffers in use, floor {Floor} buffers, released {Released}, refused: budget {BudgetRefusals}, at max {CapRefusals}, ceiling {CeilingRefusals}",
|
"Send buffer tiers: {Capacity} bytes of tier capacity, {InUse} buffers in use, floor {Floor} buffers, released {Released}, refused: budget {BudgetRefusals}, at max {CapRefusals}, ceiling {CeilingRefusals}, base {BaseCapacity} bytes, released {BaseReleased}",
|
||||||
stats.TierCapacityBytes,
|
stats.TierCapacityBytes,
|
||||||
stats.TierInUse,
|
stats.TierInUse,
|
||||||
stats.TierRetainFloor,
|
stats.TierRetainFloor,
|
||||||
stats.BuffersReleased,
|
stats.BuffersReleased,
|
||||||
stats.GrowthRefusals,
|
stats.GrowthRefusals,
|
||||||
capRefusals,
|
capRefusals,
|
||||||
ceilingRefusals
|
ceilingRefusals,
|
||||||
|
stats.BaseCapacityBytes,
|
||||||
|
stats.BaseBuffersReleased
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Slabs each base pool is divided into, after the transport applies its minimum slab size.
|
||||||
|
/// </summary>
|
||||||
|
internal static int BasePoolSlabCount(int maxBufferSlabs) =>
|
||||||
|
RingSocketManager.BasePoolSlabCount(MaxConnections, maxBufferSlabs);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Clamps the base-pool slab divisor. Fewer than one slab is meaningless, and more slabs than
|
||||||
|
/// connections cannot make a slab any smaller.
|
||||||
|
/// </summary>
|
||||||
|
internal static int CoerceMaxBufferSlabs(int configured)
|
||||||
|
{
|
||||||
|
var slabs = Math.Clamp(configured, 1, MaxConnections);
|
||||||
|
|
||||||
|
if (slabs != configured)
|
||||||
|
{
|
||||||
|
logger.Warning(
|
||||||
|
"network.maxBufferSlabs {Configured} is outside 1..{Maximum}; using {Adjusted}",
|
||||||
|
configured,
|
||||||
|
MaxConnections,
|
||||||
|
slabs
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return slabs;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Clamps the slabs of each base pool held from boot; a pool never holds more slabs than it has.
|
||||||
|
/// </summary>
|
||||||
|
internal static int CoerceInitialBufferSlabs(int configured, int slabCount)
|
||||||
|
{
|
||||||
|
var slabs = Math.Clamp(configured, 1, slabCount);
|
||||||
|
|
||||||
|
if (slabs != configured)
|
||||||
|
{
|
||||||
|
logger.Warning(
|
||||||
|
"network.initialBufferSlabs {Configured} is outside 1..{Maximum}; using {Adjusted}",
|
||||||
|
configured,
|
||||||
|
slabCount,
|
||||||
|
slabs
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return slabs;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Reads the configured send buffer size, coerced to a power of two of at least the platform
|
/// Reads the configured send buffer size, coerced to a power of two of at least the platform
|
||||||
/// allocation granularity. IORingBuffer requires this and would otherwise throw at socket
|
/// allocation granularity. IORingBuffer requires this and would otherwise throw at socket
|
||||||
|
|
|
||||||
|
|
@ -227,7 +227,39 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
|
||||||
public IAccount Account
|
public IAccount Account
|
||||||
{
|
{
|
||||||
get => _account;
|
get => _account;
|
||||||
set => _account = value;
|
set
|
||||||
|
{
|
||||||
|
_account = value;
|
||||||
|
|
||||||
|
// 0x91 credentials just verified: the character list and the world-entry burst follow.
|
||||||
|
// The login-server pass stays on the initial buffers; it never sends more than a server list.
|
||||||
|
// An off-loop password check can land its verdict after HandleReceive already flipped the
|
||||||
|
// state to LoggedIn, so both states promote; both calls below are no-ops once applied.
|
||||||
|
if (value != null && _protocolState is ProtocolState.GameServer_AwaitingGameServerLogin or ProtocolState.GameServer_LoggedIn)
|
||||||
|
{
|
||||||
|
PromoteBuffers();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void PromoteBuffers()
|
||||||
|
{
|
||||||
|
if (_socket == null)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!_socketManager.TryPromoteSendBuffer(_socket) && _socket.SendBuffer.PhysicalSize < SendBufferSize)
|
||||||
|
{
|
||||||
|
// The send path promotes on demand and disconnects if that fails too
|
||||||
|
logger.Debug("{NetState}: send buffer promotion deferred to the send path", this);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!_socketManager.TryPromoteRecvBuffer(_socket) && _socket.RecvBuffer.PhysicalSize < RecvBufferSize)
|
||||||
|
{
|
||||||
|
// The oversize-packet guard in HandlePacket gets one more try where the small buffer matters
|
||||||
|
logger.Debug("{NetState}: recv buffer promotion deferred", this);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public string Assistant { get; set; }
|
public string Assistant { get; set; }
|
||||||
|
|
@ -525,6 +557,19 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (_socket.SendBuffer.PhysicalSize < SendBufferSize)
|
||||||
|
{
|
||||||
|
// Promotion is unbudgeted and is not growth, so neither the ceiling nor the shrink
|
||||||
|
// bookkeeping applies. Before credentials verify nothing promotes: the 4 KiB ring is the
|
||||||
|
// whole pre-auth send budget, and a connection that exceeds it is dropped as exhausted.
|
||||||
|
if (_account == null)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return _socketManager.TryPromoteSendBuffer(_socket);
|
||||||
|
}
|
||||||
|
|
||||||
if (!UnderMemoryCeiling())
|
if (!UnderMemoryCeiling())
|
||||||
{
|
{
|
||||||
_ceilingRefusals++;
|
_ceilingRefusals++;
|
||||||
|
|
@ -1084,6 +1129,13 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A completion that fills the buffer arms no receive; whatever the loop consumed is free
|
||||||
|
// space again. No-op while a receive is armed or the buffer is still full.
|
||||||
|
if (_running)
|
||||||
|
{
|
||||||
|
_socket.ResumeReceive();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
|
|
@ -1137,6 +1189,22 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Can never complete: the buffer holds PhysicalSize - 1 bytes (one slot tells full from
|
||||||
|
// empty) and a recv arms only into free space
|
||||||
|
if (packetLength >= _socket.RecvBuffer.PhysicalSize)
|
||||||
|
{
|
||||||
|
// A verified account whose promotion could not be applied earlier gets one more try here,
|
||||||
|
// where the small buffer actually matters; the swap lands before the next completion's event
|
||||||
|
if (_account != null && _socket.RecvBuffer.PhysicalSize < RecvBufferSize &&
|
||||||
|
_socketManager.TryPromoteRecvBuffer(_socket))
|
||||||
|
{
|
||||||
|
return ParserState.AwaitingPartialPacket;
|
||||||
|
}
|
||||||
|
|
||||||
|
LogInfo($"Received packet 0x{packetId:X2} declaring {packetLength} bytes, more than the receive buffer holds.");
|
||||||
|
return ParserState.Error;
|
||||||
|
}
|
||||||
|
|
||||||
// Not enough data, let's wait for more to come in
|
// Not enough data, let's wait for more to come in
|
||||||
if (length < packetLength)
|
if (length < packetLength)
|
||||||
{
|
{
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,7 @@
|
||||||
</Target>
|
</Target>
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\Logger\Logger.csproj" />
|
<ProjectReference Include="..\Logger\Logger.csproj" />
|
||||||
<PackageReference Include="IORingGroup" Version="1.0.12" />
|
<PackageReference Include="IORingGroup" Version="1.0.13" />
|
||||||
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
|
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
|
||||||
<PackageReference Include="LibDeflate.Bindings" Version="1.0.4" />
|
<PackageReference Include="LibDeflate.Bindings" Version="1.0.4" />
|
||||||
<PackageReference Include="System.IO.Hashing" Version="10.0.12" />
|
<PackageReference Include="System.IO.Hashing" Version="10.0.12" />
|
||||||
|
|
|
||||||
|
|
@ -68,13 +68,34 @@ Optional systems can add substantially more. The pathfinding prebake
|
||||||
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
|
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
|
||||||
small hosts.
|
small hosts.
|
||||||
|
|
||||||
Network buffers are 64 KB receive plus a configurable send buffer per connection. Send memory is
|
Network buffers come from four pools that grow and shrink with the population rather than being
|
||||||
`network.sendBufferSize` at rest and can grow to `network.sendBufferMaxSize` under load. Shared
|
sized for a full shard. A connection that has not yet presented valid credentials holds a 4 KB
|
||||||
send-buffer tier memory is capped by `network.sendBufferGrowthBudget`, and growth is refused when
|
receive and a 4 KB send buffer (the platform's page size). On Windows Server 2012 R2 / 2016 the
|
||||||
process memory exceeds `network.memoryCeilingPercent` of available memory. The worst case is the
|
transport's legacy mapping path floors at 64 KB: the pre-auth receive pool is off there (its base is
|
||||||
base send-buffer size times the connection count, plus the shared growth budget: at the defaults,
|
64 KB), while the pre-auth send buffer starts at 64 KB under the 256 KB base. Everything the server
|
||||||
100 players is roughly 32 MB at rest, and the growth budget can add up to another 256 MB under
|
sends before that must fit in that ring — a connection that overruns it is dropped; the stock login
|
||||||
load.
|
sequence uses under 2 KB. Otherwise, when the game server verifies the account the connection is
|
||||||
|
promoted to a 64 KB receive buffer and a `network.sendBufferSize` send buffer from the base pools,
|
||||||
|
and nothing ever moves back. A flood of unauthenticated connections tops out at about 32 MB across
|
||||||
|
the full 4096-connection cap where the platform minimum is 4 KB (the transport's retained slabs and
|
||||||
|
the base pools used by logged-in players are separate), and never allocates a base-pool slab.
|
||||||
|
At boot the network holds `network.initialBufferSlabs` slab(s) of each pool — at the defaults one
|
||||||
|
2 MB receive slab, one 8 MB send slab and two 128 KB pre-auth slabs, about 10 MB — and allocates
|
||||||
|
another slab only when the population needs one. Each slab covers 32 connections at the
|
||||||
|
4096-connection maximum. After 15 quiet minutes idle slabs are trimmed back towards current usage,
|
||||||
|
never past the last 15 minutes' peak, at one slab per pool per minute and never below
|
||||||
|
`network.initialBufferSlabs`. Only the newest slab is trimmed, and buffers are handed out from the
|
||||||
|
oldest slab first, so ordinary churn empties the newest slabs; a shard that drops from 4096 players
|
||||||
|
to a handful takes about two hours to shrink fully, longer if a long-lived connection still holds a
|
||||||
|
buffer in a newer slab.
|
||||||
|
|
||||||
|
Send memory per authenticated connection is `network.sendBufferSize` at rest and can grow to
|
||||||
|
`network.sendBufferMaxSize` under load. Shared send-buffer tier memory is capped by
|
||||||
|
`network.sendBufferGrowthBudget`, and growth is refused when process memory exceeds
|
||||||
|
`network.memoryCeilingPercent` of available memory. The worst case is the receive and base
|
||||||
|
send-buffer sizes times the number of logged-in connections, plus the shared growth budget: a full
|
||||||
|
4096 logged-in connections is roughly 1.25 GB of base buffers, and the growth budget can add up to
|
||||||
|
another 256 MB.
|
||||||
|
|
||||||
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
|
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
|
||||||
Server GC on a 2-core box.
|
Server GC on a 2-core box.
|
||||||
|
|
@ -118,6 +139,8 @@ See the README for the full supported list. Two things are worth calling out:
|
||||||
| `network.sendBufferMaxSize` | 2 MB (`2097152`) | Ceiling a single connection's send buffer can grow to under load. Lower it on memory-constrained hosts; raise it if slow clients are disconnected with "send buffer exhausted". |
|
| `network.sendBufferMaxSize` | 2 MB (`2097152`) | Ceiling a single connection's send buffer can grow to under load. Lower it on memory-constrained hosts; raise it if slow clients are disconnected with "send buffer exhausted". |
|
||||||
| `network.sendBufferGrowthBudget` | 256 MB (`268435456`) | Cap on the shared memory the larger send-buffer tiers may use. Lower it on memory-constrained hosts. |
|
| `network.sendBufferGrowthBudget` | 256 MB (`268435456`) | Cap on the shared memory the larger send-buffer tiers may use. Lower it on memory-constrained hosts. |
|
||||||
| `network.memoryCeilingPercent` | 80% | Refuse send-buffer growth once the process is above this share of available memory; 0 turns the check off. |
|
| `network.memoryCeilingPercent` | 80% | Refuse send-buffer growth once the process is above this share of available memory; 0 turns the check off. |
|
||||||
|
| `network.initialBufferSlabs` | `1` | Slabs of each base pool held from boot, and the floor the trim never goes below. Raise it on a large shard to pre-warm the pools instead of paying for a slab as the population climbs. |
|
||||||
|
| `network.maxBufferSlabs` | `128` | Divides the connection maximum into base-pool slabs: a slab holds `MaxConnections / maxBufferSlabs` connections, 32 at the default. Raise it for finer slabs on a small host (the slab floor is 16 buffers); lowering it makes each slab, and the boot allocation, larger. It is not a connection or memory cap — both pools still reach the connection maximum. |
|
||||||
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
|
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
|
||||||
|
|
||||||
## Am I undersized?
|
## Am I undersized?
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue