warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(network): grow the send buffer on demand instead of disconnecting (#2639)

Browse source 
## Problem

A connection's send buffer is a fixed 256 KB. A burst of world traffic (a crowded area, a mass spawn, a war) that outruns the client's acknowledgements fills it, `NetState.Send()` reports "send buffer exhausted", and the player is disconnected. Raising the size for everyone multiplies the per-connection footprint (4096 × 256 KB is already 1 GB at full occupancy, page-locked on Windows).

## What changes

- **Growth.** When a packet does not fit (the write span is too small, the packet is larger than the span, or compression returns 0), `Send()` asks the transport to grow the buffer to the next power-of-two tier and retries, up to `network.sendBufferMaxSize` (2 MB). Compression retries once per tier since its output size is not known in advance, including when the buffer is completely full. Only when growth is refused does the existing exhaustion disconnect run. The success path is unchanged.
- **Memory ceiling.** Growth is refused (with a once-a-minute warning) when the process working set exceeds `network.memoryCeilingPercent` (80) of the memory available to the process (container-aware; `0` turns the check off). The figure is sampled at startup and refreshed each maintenance tick.
- **Shrink.** A grown socket returns to the base buffer once it is drained and 30 s have passed since its last growth, attempted from the `DataSent` handler and from the 5 s alive sweep.
- **Retention.** Every minute a timer calls the transport's `Maintain()`, which trims idle tier slabs down to the peak concurrent usage of the last 15 minutes, so recurring bursts reuse buffers without allocation while rare ones give the memory back. The line logs at Debug, and only when capacity, usage, or the floor changed or a growth was refused (budget, at max, or ceiling), so an idle shard logs nothing.
- **Budget.** `network.sendBufferGrowthBudget` (256 MB) caps the tier pools' capacity; a positive value below one tier slab is raised with a warning, a negative one is clamped to 0 (growth off). Worst case is base × connections plus the budget.
- Settings are coerced with accurate warnings (power of two, minimum, 256 MB transport ceiling). `[dumpnetstates` gains the send buffer size. `dev-docs/server-requirements.md` describes the new memory story.

## Tests

`NetStateSendBufferTests` (real loopback sockets): growth instead of disconnect with a byte-exact stream, compressed growth against the compressor's own output, the grow-then-copy path, growth with a send genuinely in flight, refusal past the maximum, refusal under the ceiling, refusal on a closing socket, shrink after the hold (direct and through the alive sweep), and the setting coercions. Server.Tests 891 passed, UOContent.Tests 1052 passed against the published 1.0.12.

Reviewed per task, whole-branch, and adversarially by a second model (twice, the second time jointly with the transport branch); all findings addressed.
This commit is contained in:
Kamron Batman 2026-09-12 16:35:06 -07:00 • committed by GitHub
parent c02909e2c8
commit 31cd19b05b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 761 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

175
Projects/Server/Network/NetState/NetState.Network.cs
View file

@ -32,8 +32,26 @@ public partial class NetState
private const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
private const int DefaultSendBufferSize = 1024 * 256; // 256KB send buffers
private const int MinSendBufferSize = 1024 * 64; // Platform allocation granularity
private const int DefaultMaxSendBufferSize = 1024 * 1024 * 2; // 2 MB
private const long DefaultSendBufferGrowthBudget = 1024L * 1024 * 256; // 256 MB
private const int DefaultMemoryCeilingPercent = 80;
// Transport ceiling; larger values overflow its tier enumeration
private const int TransportMaxSendBufferSize = 1024 * 1024 * 256; // 256 MB
private const int MaxConnections = 4096; // Max concurrent connections
internal static int MaxSendBufferSize { get; private set; }
private static long _sendBufferGrowthBudget;
private static int _memoryCeilingPercent;
// Refreshed by the maintenance sweep; internal for tests
internal static long _availableMemoryBytes;
private static Timer.DelayCallTimer _maintenanceTimer;
private static long _lastTierCapacityBytes;
private static int _lastTierInUse;
private static int _lastTierRetainFloor;
private static readonly Queue<NetState> _disposed = [];
private static readonly TimeSpan ConnectingSocketIdleLimit = TimeSpan.FromMilliseconds(5000); // 5 seconds
@ -121,16 +139,26 @@ public partial class NetState
// latency is roughly completion RTT / this value.
var maxOutstandingSends = ServerConfiguration.GetOrUpdateSetting("network.maxOutstandingSends", 32);
// Initialize IORingGroup
var ring = IORingGroup.Create(
queueSize: MaxConnections * 2,
maxConnections: MaxConnections,
maxOutstandingSends: maxOutstandingSends
);
// Per-connection send buffer: the lever for "send buffer exhausted" disconnects, and the
// per-connection memory ceiling.
var sendBufferSize = GetSendBufferSize();
MaxSendBufferSize = GetPowerOfTwoSetting("network.sendBufferMaxSize", DefaultMaxSendBufferSize, sendBufferSize);
_sendBufferGrowthBudget = CoerceSendBufferGrowthBudget(
ServerConfiguration.GetOrUpdateSetting("network.sendBufferGrowthBudget", DefaultSendBufferGrowthBudget),
sendBufferSize,
MaxSendBufferSize
);
// 0 disables the ceiling
_memoryCeilingPercent = Math.Clamp(ServerConfiguration.GetOrUpdateSetting("network.memoryCeilingPercent", DefaultMemoryCeilingPercent), 0, 100);
_availableMemoryBytes = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes;
const int maxBufferSlabs = 32;
var ring = IORingGroup.Create(
queueSize: MaxConnections * 2,
maxConnections: MaxConnections,
maxOutstandingSends: maxOutstandingSends,
maxRegisteredBuffers: RingSocketManager.RequiredRegisteredBuffers(MaxConnections, sendBufferSize, MaxSendBufferSize, _sendBufferGrowthBudget, maxBufferSlabs)
);
// Create socket manager which handles buffer pools and socket lifecycle
_socketManager = new RingSocketManager(
@ -139,8 +167,51 @@ public partial class NetState
recvBufferSize: RecvBufferSize,
sendBufferSize: sendBufferSize,
initialBufferSlabs: 8,
maxBufferSlabs: 32
maxBufferSlabs: maxBufferSlabs,
maxSendBufferSize: MaxSendBufferSize,
sendBufferGrowthBudget: _sendBufferGrowthBudget
);
_maintenanceTimer = Timer.DelayCall(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1), MaintainSendBuffers);
}
internal static void MaintainSendBuffers()
{
if (_socketManager == null)
{
return;
}
// Container limits can change
_availableMemoryBytes = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes;
var ceilingRefusals = _ceilingRefusals;
var capRefusals = _capRefusals;
_ceilingRefusals = 0;
_capRefusals = 0;
var stats = _socketManager.Maintain();
var changed = stats.TierCapacityBytes != _lastTierCapacityBytes ||
stats.TierInUse != _lastTierInUse ||
stats.TierRetainFloor != _lastTierRetainFloor;
_lastTierCapacityBytes = stats.TierCapacityBytes;
_lastTierInUse = stats.TierInUse;
_lastTierRetainFloor = stats.TierRetainFloor;
// Quiet unless something moved
if (changed || stats.BuffersReleased > 0 || stats.GrowthRefusals > 0 || capRefusals > 0 || ceilingRefusals > 0)
{
logger.Debug(
"Send buffer tiers: {Capacity} bytes of tier capacity, {InUse} buffers in use, floor {Floor} buffers, released {Released}, refused: budget {BudgetRefusals}, at max {CapRefusals}, ceiling {CeilingRefusals}",
stats.TierCapacityBytes,
stats.TierInUse,
stats.TierRetainFloor,
stats.BuffersReleased,
stats.GrowthRefusals,
capRefusals,
ceilingRefusals
);
}
}
/// <summary>
@ -148,29 +219,89 @@ public partial class NetState
/// allocation granularity. IORingBuffer requires this and would otherwise throw at socket
/// creation rather than at startup.
/// </summary>
private static int GetSendBufferSize()
private static int GetSendBufferSize() =>
GetPowerOfTwoSetting("network.sendBufferSize", DefaultSendBufferSize, MinSendBufferSize);
private static int GetPowerOfTwoSetting(string key, int defaultValue, int minimum) =>
CoercePowerOfTwoSetting(key, ServerConfiguration.GetOrUpdateSetting(key, defaultValue), minimum);
/// <summary>
/// Clamps to a power of two between <paramref name="minimum"/> and the transport ceiling.
/// </summary>
internal static int CoercePowerOfTwoSetting(string key, int configured, int minimum)
{
var configured = ServerConfiguration.GetOrUpdateSetting("network.sendBufferSize", DefaultSendBufferSize);
var size = Math.Max(MinSendBufferSize, configured);
var size = configured;
if (size > TransportMaxSendBufferSize)
{
logger.Warning(
"{Key} {Configured} is above the transport maximum {Maximum} (capped); using {Adjusted}",
key,
configured,
TransportMaxSendBufferSize,
TransportMaxSendBufferSize
);
size = TransportMaxSendBufferSize;
}
if (size < minimum)
{
logger.Warning(
"{Key} {Configured} is below the minimum {Minimum} (raised); using {Adjusted}",
key,
configured,
minimum,
minimum
);
size = minimum;
}
if (!BitOperations.IsPow2(size))
{
size = (int)BitOperations.RoundUpToPowerOf2((uint)size);
}
// Rounding up cannot cross the ceiling, itself a power of two
var rounded = (int)BitOperations.RoundUpToPowerOf2((uint)size);
if (size != configured)
{
logger.Warning(
"network.sendBufferSize {Configured} is not a power of two of at least {Minimum}; using {Adjusted}",
configured,
MinSendBufferSize,
size
);
logger.Warning("{Key} {Configured} is not a power of two; using {Adjusted}", key, configured, rounded);
size = rounded;
}
return size;
}
/// <summary>
/// Negative disables growth; below one tier slab is raised to the minimum.
/// </summary>
internal static long CoerceSendBufferGrowthBudget(long configured, int sendBufferSize, int maxSendBufferSize)
{
if (configured < 0)
{
logger.Warning("network.sendBufferGrowthBudget {Configured} is negative; using 0 (growth disabled)", configured);
return 0;
}
if (configured == 0 || maxSendBufferSize <= sendBufferSize)
{
return configured;
}
// Tier buffers are allocated a slab at a time.
var minimumBudget = RingSocketManager.MinimumSendBufferGrowthBudget(sendBufferSize);
if (configured < minimumBudget)
{
logger.Warning(
"network.sendBufferGrowthBudget {Configured} is below one tier slab; using {Minimum}",
configured,
minimumBudget
);
return minimumBudget;
}
return configured;
}
/// <summary>
/// Starts the network server on configured listening addresses.
/// </summary>
@ -494,6 +625,7 @@ public partial class NetState
{
// Update activity check on successful send
nsSend.NextActivityCheck = curTicks + 30000;
nsSend.TryShrinkSendBuffer(curTicks);
}
break;
}
@ -625,6 +757,7 @@ public partial class NetState
foreach (var ns in Instances)
{
ns.CheckAlive(curTicks);
ns.TryShrinkSendBuffer(curTicks);
}
}
catch (Exception ex)