warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(network): grow the send buffer on demand instead of disconnecting (#2639)

Browse source 
## Problem

A connection's send buffer is a fixed 256 KB. A burst of world traffic (a crowded area, a mass spawn, a war) that outruns the client's acknowledgements fills it, `NetState.Send()` reports "send buffer exhausted", and the player is disconnected. Raising the size for everyone multiplies the per-connection footprint (4096 × 256 KB is already 1 GB at full occupancy, page-locked on Windows).

## What changes

- **Growth.** When a packet does not fit (the write span is too small, the packet is larger than the span, or compression returns 0), `Send()` asks the transport to grow the buffer to the next power-of-two tier and retries, up to `network.sendBufferMaxSize` (2 MB). Compression retries once per tier since its output size is not known in advance, including when the buffer is completely full. Only when growth is refused does the existing exhaustion disconnect run. The success path is unchanged.
- **Memory ceiling.** Growth is refused (with a once-a-minute warning) when the process working set exceeds `network.memoryCeilingPercent` (80) of the memory available to the process (container-aware; `0` turns the check off). The figure is sampled at startup and refreshed each maintenance tick.
- **Shrink.** A grown socket returns to the base buffer once it is drained and 30 s have passed since its last growth, attempted from the `DataSent` handler and from the 5 s alive sweep.
- **Retention.** Every minute a timer calls the transport's `Maintain()`, which trims idle tier slabs down to the peak concurrent usage of the last 15 minutes, so recurring bursts reuse buffers without allocation while rare ones give the memory back. The line logs at Debug, and only when capacity, usage, or the floor changed or a growth was refused (budget, at max, or ceiling), so an idle shard logs nothing.
- **Budget.** `network.sendBufferGrowthBudget` (256 MB) caps the tier pools' capacity; a positive value below one tier slab is raised with a warning, a negative one is clamped to 0 (growth off). Worst case is base × connections plus the budget.
- Settings are coerced with accurate warnings (power of two, minimum, 256 MB transport ceiling). `[dumpnetstates` gains the send buffer size. `dev-docs/server-requirements.md` describes the new memory story.

## Tests

`NetStateSendBufferTests` (real loopback sockets): growth instead of disconnect with a byte-exact stream, compressed growth against the compressor's own output, the grow-then-copy path, growth with a send genuinely in flight, refusal past the maximum, refusal under the ceiling, refusal on a closing socket, shrink after the hold (direct and through the alive sweep), and the setting coercions. Server.Tests 891 passed, UOContent.Tests 1052 passed against the published 1.0.12.

Reviewed per task, whole-branch, and adversarially by a second model (twice, the second time jointly with the transport branch); all findings addressed.
This commit is contained in:
Kamron Batman 2026-09-12 16:35:06 -07:00 • committed by GitHub
parent c02909e2c8
commit 31cd19b05b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 761 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

153
Projects/Server/Network/NetState/NetState.cs
View file

@ -47,6 +47,20 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
private static readonly HashSet<NetState> _instances = new(2048);
public static HashSet<NetState> Instances => _instances;
// GC's container-aware figure: a heuristic, not a hard bound; fails open when unpopulated
private static bool UnderMemoryCeiling() =>
_memoryCeilingPercent <= 0 ||
_availableMemoryBytes <= 0 ||
Environment.WorkingSet < _availableMemoryBytes / 100 * _memoryCeilingPercent;
private const long MemoryCeilingWarnIntervalMs = 60000;
private static long _memoryCeilingWarnedAt;
private static bool _memoryCeilingWarned;
// Reset by MaintainSendBuffers; the transport counts budget refusals
private static int _ceilingRefusals;
private static int _capRefusals;
private readonly string _toString;
private ClientVersion _version;
private bool _running = true;
@ -59,6 +73,10 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
private long _drainDeadline;
private bool _drainDeadlineArmed;
internal bool _sendBufferGrown;
internal long _sendBufferGrewAt;
internal const long SendBufferHoldMs = 30000;
internal ParserState _parserState = ParserState.AwaitingNextPacket;
internal ProtocolState _protocolState = ProtocolState.AwaitingSeed;
private bool _packetLogging;
@ -480,6 +498,83 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return buffer.Length > 0;
}
// Grows tier by tier until `needed` fits or a tier is refused.
internal bool TryGrowSendBuffer(int needed)
{
if (_socket == null)
{
return false;
}
while (_socket.SendBuffer.WritableBytes < needed)
{
if (!TryGrowSendBufferOneTier())
{
return false;
}
}
return true;
}
// One tier; compression learns its output size by retrying
internal bool TryGrowSendBufferOneTier()
{
if (_socket == null)
{
return false;
}
if (!UnderMemoryCeiling())
{
_ceilingRefusals++;
var now = Core.TickCount;
if (!_memoryCeilingWarned || now - (_memoryCeilingWarnedAt + MemoryCeilingWarnIntervalMs) >= 0)
{
_memoryCeilingWarned = true;
_memoryCeilingWarnedAt = now;
logger.Warning("Send buffer growth refused: process is above {Percent}% of available memory", _memoryCeilingPercent);
}
return false;
}
if (!_socketManager.TryGrowSendBuffer(_socket))
{
// At the cap; the transport counts budget refusals
if (_socket.SendBuffer.PhysicalSize >= MaxSendBufferSize)
{
_capRefusals++;
}
return false;
}
_sendBufferGrown = true;
_sendBufferGrewAt = Core.TickCount;
logger.Debug("{NetState}: send buffer grown to {Size}", this, _socket.SendBuffer.PhysicalSize);
return true;
}
// The pool retains the larger buffer
internal bool TryShrinkSendBuffer(long curTicks)
{
if (!_sendBufferGrown || _socket == null || curTicks - (_sendBufferGrewAt + SendBufferHoldMs) < 0)
{
return false;
}
if (!_socketManager.TryShrinkSendBuffer(_socket))
{
return false;
}
_sendBufferGrown = false;
logger.Debug("{NetState}: send buffer returned to {Size}", this, _socket.SendBuffer.PhysicalSize);
return true;
}
public void Send(ReadOnlySpan<byte> span)
{
if (span == ReadOnlySpan<byte>.Empty || this.CannotSendPackets())
@ -499,31 +594,57 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return;
}
// Never drop silently: the client would stay connected while missing game state.
if (!GetSendBuffer(out var buffer))
{
SendBufferExhausted(length, 0);
return;
}
try
{
// Never drop silently: the client would stay connected while missing game state.
if (!GetSendBuffer(out var buffer))
{
// Full; the compressed size is unknown, so grow one tier and let the retry loop finish
var grown = CompressionEnabled ? TryGrowSendBufferOneTier() : TryGrowSendBuffer(length);
if (!grown || !GetSendBuffer(out buffer))
{
SendBufferExhausted(length);
return;
}
}
// Apply encoding first (e.g., compression from UOContent)
if (CompressionEnabled)
{
length = NetworkCompression.Compress(span, buffer);
// 0 means nothing was written, whether it did not fit or the input was too large.
if (length <= 0)
{
SendBufferExhausted(span.Length, buffer.Length);
return;
// Compress refuses this length outright; growth cannot help
if (span.Length > NetworkCompression.DefiniteOverflow)
{
SendBufferExhausted(span.Length);
return;
}
// Output size is unknown until compressed; grow a tier and retry
while (length <= 0 && TryGrowSendBufferOneTier() && GetSendBuffer(out buffer))
{
length = NetworkCompression.Compress(span, buffer);
}
if (length <= 0)
{
SendBufferExhausted(span.Length);
return;
}
}
}
else if (span.Length > buffer.Length)
{
SendBufferExhausted(span.Length, buffer.Length);
return;
if (!TryGrowSendBuffer(span.Length) || !GetSendBuffer(out buffer))
{
SendBufferExhausted(span.Length);
return;
}
span.CopyTo(buffer);
}
else
{
@ -558,9 +679,9 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
/// </summary>
/// <remarks>
/// High unacked means a slow client holding the buffer; needed approaching capacity means the
/// buffer is too small for this shard and network.sendBufferSize should be raised.
/// buffer is too small for this shard and network.sendBufferMaxSize should be raised.
/// </remarks>
private void SendBufferExhausted(int needed, int writable)
private void SendBufferExhausted(int needed)
{
// One report per disconnect; the first reason wins
if (_disconnectQueued)
@ -568,12 +689,14 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return;
}
// Read fresh; the caller's span may predate a growth
var sendBuffer = _socket?.SendBuffer;
var writable = sendBuffer?.WritableBytes ?? 0;
var unacked = sendBuffer?.InFlightBytes ?? 0;
var capacity = sendBuffer?.PhysicalSize ?? 0;
logger.Warning(
"{NetState}: send buffer exhausted - needed {Needed} bytes, {Writable} writable, {Unacked} awaiting acknowledgement, {Capacity} capacity. Raise network.sendBufferSize (power of two) if this recurs on healthy connections.",
"{NetState}: send buffer exhausted - needed {Needed} bytes, {Writable} writable, {Unacked} awaiting acknowledgement, {Capacity} capacity. Raise network.sendBufferMaxSize (power of two) if this recurs on healthy connections.",
this,
needed,
writable,