fix: Fixes send-path backpressure: consume IORingGroup 1.0.8, stop dropping packets silently (#2551)
## Summary Two related fixes on the outbound path: 1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it. 2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state. ## 1. Send-path stall (RIO) RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued. Measured on a 50ms-RTT production shard: | | before | after | |---|---|---| | in-game latency, data flowing | **101–146 ms** | **48–51 ms** | | p95 | ~135 ms | 52.8 ms | | samples > 70 ms | 20 | **0** | The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did. Two things worth recording, because they explain why this went unnoticed: - As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones. - The same measurement at loopback RTT is **microseconds**, so local testing could never surface it. New settings, both restart-time: - **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`. - **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling. ## 2. Send buffer full `NetState.Send` had three failure modes once the buffer filled, none of them visible: | writable | behaviour | |---|---| | `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect | | `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way | | `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** | The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space. `Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`. ## Testing `NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical. Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8. ## Notes for reviewers - Upstream change: modernuo/IORingGroup#9. - The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise. - Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
This commit is contained in:
parent
c909ed1f2f
commit
294dcd94a0
5 changed files with 186 additions and 9 deletions
|
|
@ -19,6 +19,7 @@ using System.Linq;
|
|||
using System.Net;
|
||||
using System.Net.NetworkInformation;
|
||||
using System.Network;
|
||||
using System.Numerics;
|
||||
|
||||
namespace Server.Network;
|
||||
|
||||
|
|
@ -28,9 +29,10 @@ namespace Server.Network;
|
|||
public partial class NetState
|
||||
{
|
||||
// Buffer sizes
|
||||
private const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
|
||||
private const int SendBufferSize = 1024 * 256; // 256KB send buffers
|
||||
private const int MaxConnections = 4096; // Max concurrent connections
|
||||
private const int RecvBufferSize = 1024 * 64; // 64KB recv buffers
|
||||
private const int DefaultSendBufferSize = 1024 * 256; // 256KB send buffers
|
||||
private const int MinSendBufferSize = 1024 * 64; // Platform allocation granularity
|
||||
private const int MaxConnections = 4096; // Max concurrent connections
|
||||
|
||||
private static readonly Queue<NetState> _disposed = [];
|
||||
private static readonly TimeSpan ConnectingSocketIdleLimit = TimeSpan.FromMilliseconds(5000); // 5 seconds
|
||||
|
|
@ -41,7 +43,9 @@ public partial class NetState
|
|||
// NetState storage indexed by RingSocket.Id
|
||||
private static readonly NetState[] _netStates = new NetState[MaxConnections];
|
||||
|
||||
// Events buffer for ProcessCompletions
|
||||
// Events buffer for ProcessCompletions. Bounded by one event per peeked completion
|
||||
// (maxSockets), doubled for headroom. Undersizing drops DataReceived events whose bytes were
|
||||
// already committed, leaving them unparsed until the next recv completes.
|
||||
private static readonly RingSocketEvent[] _events = new RingSocketEvent[MaxConnections * 2];
|
||||
|
||||
// Listener management
|
||||
|
|
@ -88,20 +92,61 @@ public partial class NetState
|
|||
// Initialize IP rate limiter
|
||||
_ipRateLimiter = new IPRateLimiter(10, 10000, 1000, 2.0, 3_600_000, Core.ClosingTokenSource.Token);
|
||||
|
||||
// Sends in flight per connection; honoured by RIO only (see IIORingGroup). Costs a
|
||||
// request-queue and completion-queue slot per send, not another buffer. Worst-case added
|
||||
// latency is roughly completion RTT / this value.
|
||||
var maxOutstandingSends = ServerConfiguration.GetOrUpdateSetting("network.maxOutstandingSends", 32);
|
||||
|
||||
// Initialize IORingGroup
|
||||
var ring = IORingGroup.Create(queueSize: MaxConnections * 2, maxConnections: MaxConnections);
|
||||
var ring = IORingGroup.Create(
|
||||
queueSize: MaxConnections * 2,
|
||||
maxConnections: MaxConnections,
|
||||
maxOutstandingSends: maxOutstandingSends
|
||||
);
|
||||
|
||||
// Per-connection send buffer: the lever for "send buffer exhausted" disconnects, and the
|
||||
// per-connection memory ceiling.
|
||||
var sendBufferSize = GetSendBufferSize();
|
||||
|
||||
// Create socket manager which handles buffer pools and socket lifecycle
|
||||
_socketManager = new RingSocketManager(
|
||||
ring,
|
||||
maxSockets: MaxConnections,
|
||||
recvBufferSize: RecvBufferSize,
|
||||
sendBufferSize: SendBufferSize,
|
||||
sendBufferSize: sendBufferSize,
|
||||
initialBufferSlabs: 8,
|
||||
maxBufferSlabs: 32
|
||||
);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the configured send buffer size, coerced to a power of two of at least the platform
|
||||
/// allocation granularity. IORingBuffer requires this and would otherwise throw at socket
|
||||
/// creation rather than at startup.
|
||||
/// </summary>
|
||||
private static int GetSendBufferSize()
|
||||
{
|
||||
var configured = ServerConfiguration.GetOrUpdateSetting("network.sendBufferSize", DefaultSendBufferSize);
|
||||
var size = Math.Max(MinSendBufferSize, configured);
|
||||
|
||||
if (!BitOperations.IsPow2(size))
|
||||
{
|
||||
size = (int)BitOperations.RoundUpToPowerOf2((uint)size);
|
||||
}
|
||||
|
||||
if (size != configured)
|
||||
{
|
||||
logger.Warning(
|
||||
"network.sendBufferSize {Configured} is not a power of two of at least {Minimum}; using {Adjusted}",
|
||||
configured,
|
||||
MinSendBufferSize,
|
||||
size
|
||||
);
|
||||
}
|
||||
|
||||
return size;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts the network server on configured listening addresses.
|
||||
/// </summary>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue