294dcd94a0
## Summary Two related fixes on the outbound path: 1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it. 2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state. ## 1. Send-path stall (RIO) RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued. Measured on a 50ms-RTT production shard: | | before | after | |---|---|---| | in-game latency, data flowing | **101–146 ms** | **48–51 ms** | | p95 | ~135 ms | 52.8 ms | | samples > 70 ms | 20 | **0** | The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did. Two things worth recording, because they explain why this went unnoticed: - As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones. - The same measurement at loopback RTT is **microseconds**, so local testing could never surface it. New settings, both restart-time: - **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`. - **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling. ## 2. Send buffer full `NetState.Send` had three failure modes once the buffer filled, none of them visible: | writable | behaviour | |---|---| | `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect | | `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way | | `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** | The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space. `Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`. ## Testing `NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical. Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8. ## Notes for reviewers - Upstream change: modernuo/IORingGroup#9. - The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise. - Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
86 lines
2.6 KiB
C#
86 lines
2.6 KiB
C#
using System;
|
|
using Server.Network;
|
|
using Xunit;
|
|
|
|
namespace Server.Tests.Network;
|
|
|
|
/// <summary>
|
|
/// Bounds behaviour of the Huffman compressor when the destination is too small.
|
|
///
|
|
/// This is reachable in production: NetState only checks that the send buffer has *some* writable
|
|
/// space before handing the remainder to Compress, so a nearly-full buffer can offer a span of one
|
|
/// to three bytes. The internal guard is computed as an unsigned <c>output.Length - 4</c>, which
|
|
/// underflows for those sizes and stops bounding the writes at all.
|
|
/// </summary>
|
|
public class NetworkCompressionBoundsTests
|
|
{
|
|
[Theory]
|
|
[InlineData(0)]
|
|
[InlineData(1)]
|
|
[InlineData(2)]
|
|
[InlineData(3)]
|
|
public void RefusesOutputTooSmallToBound(int outputSize)
|
|
{
|
|
var input = new byte[64];
|
|
Array.Fill(input, (byte)'A');
|
|
|
|
// Sentinel-filled backing array; only the middle window is offered to the compressor, so
|
|
// any write past the span shows up as a modified sentinel rather than silent corruption.
|
|
var backing = new byte[256];
|
|
Array.Fill(backing, (byte)0xCC);
|
|
|
|
const int windowStart = 64;
|
|
var output = backing.AsSpan(windowStart, outputSize);
|
|
|
|
var written = NetworkCompression.Compress(input, output);
|
|
|
|
Assert.Equal(0, written);
|
|
|
|
for (var i = 0; i < backing.Length; i++)
|
|
{
|
|
Assert.Equal(0xCC, backing[i]);
|
|
}
|
|
}
|
|
|
|
[Fact]
|
|
public void StillCompressesWhenOutputIsLargeEnough()
|
|
{
|
|
var input = new byte[64];
|
|
Array.Fill(input, (byte)'A');
|
|
|
|
var output = new byte[256];
|
|
|
|
var written = NetworkCompression.Compress(input, output);
|
|
|
|
Assert.True(written > 0);
|
|
Assert.True(written <= output.Length);
|
|
}
|
|
|
|
[Fact]
|
|
public void ReportsFailureRatherThanOverrunningATightOutput()
|
|
{
|
|
// Large input against a small-but-bounded output: the guard is well-defined here, so this
|
|
// must fail cleanly rather than write past the end.
|
|
var input = new byte[4096];
|
|
Array.Fill(input, (byte)'A');
|
|
|
|
var backing = new byte[256];
|
|
Array.Fill(backing, (byte)0xCC);
|
|
|
|
const int windowStart = 64;
|
|
const int windowSize = 16;
|
|
var output = backing.AsSpan(windowStart, windowSize);
|
|
|
|
NetworkCompression.Compress(input, output);
|
|
|
|
for (var i = 0; i < windowStart; i++)
|
|
{
|
|
Assert.Equal(0xCC, backing[i]);
|
|
}
|
|
|
|
for (var i = windowStart + windowSize; i < backing.Length; i++)
|
|
{
|
|
Assert.Equal(0xCC, backing[i]);
|
|
}
|
|
}
|
|
}
|