fix(network): stop sending once a disconnect is handed to the socket (#2637)

## Problem

When a client's send buffer fills, the `send buffer exhausted` warning repeats for every packet, every tick, until the NetState is finally disposed. Before the io_uring transport an overflow produced one message plus the disconnect line.

## Root cause

Since #2315, `NetState.Disconnect()` only queues. The NetState keeps running, the Mobile stays attached, and in `Slice()` the queued disconnect becomes `RingSocket.Disconnect()`, which sees buffered or in-flight sends and merely sets `DisconnectPending` while the transport drains. Nothing stopped game logic from writing into that buffer afterwards, so:

- every broadcast to that player still reached `Send()`, hit the full buffer, and re-reported exhaustion (made visible by #2551);
- refills kept `ReadableBytes` above zero, so the graceful drain could never finish, and `DataSent` completions kept pushing the alive check out. A slow-but-acking client could keep a "disconnected" session attached indefinitely.

Two more sources of the same warning surfaced during the analysis: `Dispose()` sets `_running = false` before nulling `Mobile.NetState`, and the setter's bank-close / target-cancel packets then reported `0 writable`; and when the socket takes the immediate-close branch (nothing in flight) `Connected` drops without `DisconnectPending`, leaving a one-tick window that also re-reported.

## Changes

- `CannotSendPackets()` refuses once the socket is `DisconnectPending` or no longer `Connected`. Sends between `Disconnect()` and the `Slice()` handoff are still delivered (kicks with a message, the play-server ack).
- `SendBufferExhausted()` reports once per disconnect and keeps the first reason.
- `Send()` is silent while closing instead of reporting exhaustion for a socket that is going away.
- `_nextAliveCheck` is seeded from a real tick (the zero default suppressed the alive sweep on hosts whose counter starts negative).
- `CancelAllTrades()` had its null guard inverted since 547c2ea0f (#2603) and never cancelled anything.

## Does the gate cut off the graceful flush?

No. The gate only refuses writes made after `Slice()` has handed the disconnect to the socket. Everything committed before that point is drained by the transport, which then sends FIN. This matches the pre-io_uring lifecycle: `Disconnect()` cleared `_running` at once, and the next `Slice()` made its final `Flush()` and then closed the socket in `Dispose()`. In both worlds the send window after `Disconnect()` ends at the next network slice; the old one made a single flush attempt, the new one drains everything buffered.

Checked flows:

- **Login gateway.** `PlayServer` sends the 0x8C ack, and the parser queues `Disconnect()` in the same `HandleReceive` call. Both happen before the handoff, so the ack is delivered and FIN follows. ClassicUO's `HandleRelayServerPacket` disconnects and opens a fresh connection before sending the seed and second login, so the game login is a new NetState. The `LoginServer_ServerSelectAck` "CUO/Orion do not reconnect" fallback (#489, 2021) resets the parser state and returns without parsing or replying, and no further receive is posted once the disconnect is pending, so it sends nothing either way. Orion likewise opens a separate game socket before closing the login one.
- **Login rejections, character create/select/delete errors, duplicate-packet guards.** Each sends its rejection first and calls `Disconnect()` in the same handler.
- **Kicks and bans** (`[kick`, `[ban`, AdminGump, ClientGump, ClientVerification, AssistantHandler, lockdown). The message is sent first; delayed variants fire `Disconnect()` from a timer with nothing sent afterwards.
- **Main loop order.** Timers run before `NetState.Slice()`, packet handlers run inside it before the flush, and `LoopContext` tasks run after it; in every case a send that precedes `Disconnect()` reaches the buffer before the next handoff.

`Send_BeforeDisconnect_IsDeliveredThenPeerSeesEof` and `Send_LargeBeforeDisconnect_IsFullyDrainedThenPeerSeesEof` read the bytes back from the peer socket after the handoff (the latter 192 KB across several send completions, past the loopback kernel buffers) and then wait for the FIN, proving delivery and clean close with the gate in place.

## Tests

Seven new tests in `NetStateDisconnectTests` over real loopback sockets: force-close after the drain deadline, delivery then EOF for small and multi-completion sends before `Disconnect()`, send dropped after the handoff, send dropped after an immediate close, exhaustion reported once with the first reason kept, and trades cancelled on disconnect. `MockAccount` promoted to a shared test helper. Server.Tests 876 passed, UOContent.Tests 1048 passed against the published 1.0.11.

## Drain deadline and IORingGroup 1.0.11

A socket handed a disconnect drains what is buffered and closes once the peer has acknowledged it. Send completions keep `NextActivityCheck` moving, so a slow but acking peer could hold a closing socket open indefinitely. A deadline (`DrainTimeoutMs`, 10 s) is now armed at the handoff, or on first sight of a transport-initiated drain in `CheckAlive`, and force-closes when it passes, independent of the inactivity check.

That force-close is only safe with IORingGroup 1.0.11 (modernuo/IORingGroup#12), which this PR bumps to. Before it, `DisconnectImmediate` released pooled buffers while recv/send operations could still be in flight, a failed send stranded the socket forever, and a recv completion could be delivered after its buffer was released. 1.0.11 retires every outstanding operation before release, aborts on a failed send, and holds buffers until the pass after the `Disconnected` event.
This commit is contained in:
Kamron Batman 2026-09-11 23:18:37 -07:00 committed by GitHub
parent 309fcfeb27
commit d16166591c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 393 additions and 58 deletions

View file

@ -0,0 +1,43 @@
using System;
using System.Collections.Generic;
using Server.Accounting;
namespace Server.Tests.Network;
/// <summary>
/// Minimal IAccount so a test NetState looks authenticated.
/// </summary>
public class MockAccount : IAccount
{
public int TotalGold { get; }
public int TotalPlat { get; }
public bool DepositGold(int amount) => throw new NotImplementedException();
public bool DepositPlat(int amount) => throw new NotImplementedException();
public bool WithdrawGold(int amount) => throw new NotImplementedException();
public bool WithdrawPlat(int amount) => throw new NotImplementedException();
public long GetTotalGold() => throw new NotImplementedException();
public int CompareTo(IAccount other) => throw new NotImplementedException();
public string Username { get; }
public string Email { get; set; }
public AccessLevel AccessLevel { get; set; }
public int Length { get; }
public int Limit { get; set; } = 6; // Default to 6 character slots
public int Count { get; }
private readonly Dictionary<int, Mobile> _mobiles = new();
public Mobile this[int index]
{
get => _mobiles.GetValueOrDefault(index);
set => _mobiles[index] = value;
}
public DateTime Created { get; set; }
public Serial Serial { get; }
public void Deserialize(IGenericReader reader) => throw new NotImplementedException();
public void Serialize(IGenericWriter writer) => throw new NotImplementedException();
public bool Deleted { get; }
public void Delete() => throw new NotImplementedException();
public bool TrySetUsername(string username) => throw new NotImplementedException();
public void SetPassword(string password) => throw new NotImplementedException();
public bool CheckPassword(string password) => throw new NotImplementedException();
}

View file

@ -20,7 +20,13 @@ public static class PacketTestUtilities
/// Uses a real Socket and RingSocket with actual buffers.
/// Must be disposed after use (use 'using' statement).
/// </summary>
public static NetState CreateTestNetState()
public static NetState CreateTestNetState() => CreateTestNetState(out _);
/// <summary>
/// As <see cref="CreateTestNetState()"/>, also handing back the peer socket so a test can read what the
/// server delivered.
/// </summary>
public static NetState CreateTestNetState(out Socket client)
{
NetState.Slice(); // Process disconnects/disposes
@ -65,6 +71,7 @@ public static class PacketTestUtilities
var testSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
testSocket.Connect(IPAddress.Loopback, _testPort);
_testSocketClients.Add(testSocket);
client = testSocket;
// Slice until we have a new NetState instance added
// AcceptEx is asynchronous, so we may need to wait/retry

View file

@ -416,41 +416,6 @@ public class ClientEnumeratorTests
}
}
private class MockAccount : IAccount
{
public int TotalGold { get; }
public int TotalPlat { get; }
public bool DepositGold(int amount) => throw new NotImplementedException();
public bool DepositPlat(int amount) => throw new NotImplementedException();
public bool WithdrawGold(int amount) => throw new NotImplementedException();
public bool WithdrawPlat(int amount) => throw new NotImplementedException();
public long GetTotalGold() => throw new NotImplementedException();
public int CompareTo(IAccount other) => throw new NotImplementedException();
public string Username { get; }
public string Email { get; set; }
public AccessLevel AccessLevel { get; set; }
public int Length { get; }
public int Limit { get; set; } = 6; // Default to 6 character slots
public int Count { get; }
private readonly Dictionary<int, Mobile> _mobiles = new();
public Mobile this[int index]
{
get => _mobiles.GetValueOrDefault(index);
set => _mobiles[index] = value;
}
public DateTime Created { get; set; }
public Serial Serial { get; }
public void Deserialize(IGenericReader reader) => throw new NotImplementedException();
public void Serialize(IGenericWriter writer) => throw new NotImplementedException();
public bool Deleted { get; }
public void Delete() => throw new NotImplementedException();
public bool TrySetUsername(string username) => throw new NotImplementedException();
public void SetPassword(string password) => throw new NotImplementedException();
public bool CheckPassword(string password) => throw new NotImplementedException();
}
private static (NetState, Mobile) CreateClientWithMobile(Map map, Point3D location)
{
// Create test NetState with real socket and buffers

View file

@ -0,0 +1,272 @@
using System;
using System.Diagnostics;
using System.Net.Sockets;
using Server.Network;
using Xunit;
namespace Server.Tests.Network;
/// <summary>
/// Send-side behaviour of a NetState after its disconnect is handed to the socket.
/// </summary>
[Collection("Sequential Server Tests")]
public class NetStateDisconnectTests
{
private static NetState CreateAuthenticatedNetState()
{
var ns = PacketTestUtilities.CreateTestNetState();
ns.Account = new MockAccount(); // skips the unattached-socket sweep
return ns;
}
private static (NetState ns, Mobile m) CreateClient(string name)
{
var ns = CreateAuthenticatedNetState();
var m = new Mobile(World.NewMobile) { Name = name };
m.DefaultMobileInit();
ns.Mobile = m;
m.NetState = ns;
return (ns, m);
}
[Fact]
public void Send_AfterDisconnectHandedToSocket_DropsPacket()
{
var ns = CreateAuthenticatedNetState();
try
{
ns.Disconnect("test");
NetState.Slice(); // handoff; the in-flight recv keeps it pending
Assert.True(ns.Running);
Assert.True(ns._socket.DisconnectPending);
Assert.Equal(0, ns._socket.SendBuffer.ReadableBytes);
ns.Send([0x73, 0x00]);
Assert.True(ns.CannotSendPackets());
Assert.Equal(0, ns._socket.SendBuffer.ReadableBytes);
}
finally
{
ns.Dispose();
}
}
[Fact]
public void Send_AfterImmediateDisconnect_DropsPacket()
{
var ns = CreateAuthenticatedNetState();
// Immediate branch of RingSocket.Disconnect(): Connected drops, DisconnectPending never set,
// Disconnected event lands next Slice()
try
{
NetState.SocketManager.DisconnectImmediate(ns._socket);
Assert.True(ns.Running);
Assert.False(ns._socket.Connected);
Assert.False(ns._socket.DisconnectPending);
ns.Send([0x73, 0x00]);
Assert.True(ns.CannotSendPackets());
Assert.Equal(0, ns._socket.SendBuffer.ReadableBytes);
}
finally
{
ns.Dispose();
}
}
[Fact]
public void Send_BeforeDisconnect_IsDeliveredThenPeerSeesEof()
{
var ns = PacketTestUtilities.CreateTestNetState(out var client);
ns.Account = new MockAccount();
try
{
byte[] payload = [0x8C, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
ns.Send(payload);
ns.Disconnect("redirect");
NetState.Slice(); // flush, then handoff
Assert.True(ns._socket.DisconnectPending);
var received = new byte[payload.Length];
var total = 0;
var deadline = Stopwatch.StartNew();
while (total < payload.Length && deadline.ElapsedMilliseconds < 5000)
{
NetState.Slice();
if (client.Poll(1000, SelectMode.SelectRead))
{
var read = client.Receive(received, total, payload.Length - total, SocketFlags.None);
Assert.NotEqual(0, read);
total += read;
}
}
Assert.Equal(payload, received);
// FIN follows once the send completes
var eof = -1;
while (eof != 0 && deadline.ElapsedMilliseconds < 5000)
{
NetState.Slice();
if (client.Poll(1000, SelectMode.SelectRead))
{
eof = client.Receive(received);
}
}
Assert.Equal(0, eof);
}
finally
{
ns.Dispose();
client.Close();
}
}
[Fact]
public void Send_LargeBeforeDisconnect_IsFullyDrainedThenPeerSeesEof()
{
var ns = PacketTestUtilities.CreateTestNetState(out var client);
ns.Account = new MockAccount();
try
{
// Several posts' worth, larger than the loopback kernel buffers, so the drain spans completions
const int chunk = 32 * 1024;
var payload = new byte[6 * chunk];
new System.Random(7).NextBytes(payload);
for (var offset = 0; offset < payload.Length; offset += chunk)
{
ns.Send(payload.AsSpan(offset, chunk));
}
ns.Disconnect("redirect");
NetState.Slice(); // flush, then handoff
Assert.True(ns._socket.DisconnectPending);
var received = new byte[payload.Length];
var total = 0;
var deadline = Stopwatch.StartNew();
while (total < payload.Length && deadline.ElapsedMilliseconds < 10000)
{
NetState.Slice();
if (client.Poll(1000, SelectMode.SelectRead))
{
var read = client.Receive(received, total, payload.Length - total, SocketFlags.None);
Assert.NotEqual(0, read);
total += read;
}
}
Assert.Equal(payload.Length, total);
Assert.Equal(payload, received);
var eof = -1;
while (eof != 0 && deadline.ElapsedMilliseconds < 10000)
{
NetState.Slice();
if (client.Poll(1000, SelectMode.SelectRead))
{
eof = client.Receive(received);
}
}
Assert.Equal(0, eof);
}
finally
{
ns.Dispose();
client.Close();
}
}
[Fact]
public void PendingDisconnect_ForceClosesAfterDrainTimeout()
{
var ns = CreateAuthenticatedNetState();
try
{
ns.Disconnect("test");
NetState.Slice(); // handoff; the in-flight recv keeps it pending and arms the deadline
Assert.True(ns._socket.DisconnectPending);
var armed = Core.TickCount;
ns.CheckAlive(armed + NetState.DrainTimeoutMs - 1);
Assert.True(ns._socket.Connected);
ns.CheckAlive(armed + NetState.DrainTimeoutMs);
Assert.False(ns._socket.Connected);
}
finally
{
ns.Dispose();
}
}
[Fact]
public void SendBufferExhausted_WhileDisconnectQueued_KeepsFirstReason()
{
var ns = CreateAuthenticatedNetState();
try
{
var capacity = ns._socket.SendBuffer.PhysicalSize;
ns.Send(new byte[capacity + 1]); // cannot fit; queues the disconnect
ns.Send(new byte[capacity + 2]); // same tick; must not re-report
Assert.Contains($"needed {capacity + 1}", ns._disconnectReason);
}
finally
{
ns.Dispose();
}
}
[Fact]
public void CancelAllTrades_CancelsEveryTrade()
{
var (from, fromMobile) = CreateClient("from");
var (to, toMobile) = CreateClient("to");
try
{
from.AddTrade(to);
var trade = from.FindTrade(toMobile);
Assert.NotNull(trade);
Assert.True(trade.Valid);
from.CancelAllTrades();
Assert.False(trade.Valid);
Assert.Null(from.Trades);
Assert.Null(to.Trades);
Assert.Null(from.FindTrade(toMobile));
Assert.Null(to.FindTrade(fromMobile));
}
finally
{
from.Mobile = null;
from.Dispose();
fromMobile.Delete();
to.Mobile = null;
to.Dispose();
toMobile.Delete();
}
}
}

View file

@ -61,6 +61,9 @@ public partial class NetState
/// </summary>
public static IIORingGroup Ring => _socketManager?.Ring;
// Test hook
internal static RingSocketManager SocketManager => _socketManager;
/// <summary>
/// Waits for network I/O completions or until the specified timeout expires.
/// Used by the game loop to sleep efficiently while remaining responsive to network events.
@ -107,6 +110,9 @@ public partial class NetState
return;
}
// Seed from a real tick; a zero default suppresses the sweep when ticks start negative
_nextAliveCheck = Core.TickCount;
// Initialize IP rate limiter
_ipRateLimiter = new IPRateLimiter(10, 10000, 1000, 2.0, 3_600_000, Core.ClosingTokenSource.Token);
@ -539,6 +545,11 @@ public partial class NetState
// - Waits for in-flight I/O to complete
// - Ensures buffers aren't released while kernel is still using them
ns._socket.Disconnect();
if (ns._socket.DisconnectPending)
{
ns.ArmDrainDeadline(curTicks);
}
}
}

View file

@ -36,6 +36,7 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
private const int HuePickerCap = 512;
private const int MenuCap = 512;
private const int PacketPerSecondThreshold = 3000;
internal const long DrainTimeoutMs = 10000; // graceful disconnect gets this long to drain
private static readonly Queue<NetState> _flushPending = new(2048);
private static readonly Queue<NetState> _pendingDisconnects = new(256); // Processed AFTER flush
@ -54,7 +55,9 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
private bool _disconnectQueued; // Queued for disconnect processing (after flush)
private long[] _packetThrottles;
private long[] _packetCounts;
private string _disconnectReason = string.Empty;
internal string _disconnectReason = string.Empty;
private long _drainDeadline;
private bool _drainDeadlineArmed;
internal ParserState _parserState = ParserState.AwaitingNextPacket;
internal ProtocolState _protocolState = ProtocolState.AwaitingSeed;
@ -294,7 +297,8 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
for (var i = Trades.Count - 1; i >= 0; --i)
{
if (Trades != null)
// RemoveTrade() nulls the list once empty
if (Trades == null)
{
break;
}
@ -489,6 +493,12 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return;
}
// Closing; nothing to report
if (!_running || _socket == null)
{
return;
}
// Never drop silently: the client would stay connected while missing game state.
if (!GetSendBuffer(out var buffer))
{
@ -552,6 +562,12 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
/// </remarks>
private void SendBufferExhausted(int needed, int writable)
{
// One report per disconnect; the first reason wins
if (_disconnectQueued)
{
return;
}
var sendBuffer = _socket?.SendBuffer;
var unacked = sendBuffer?.InFlightBytes ?? 0;
var capacity = sendBuffer?.PhysicalSize ?? 0;
@ -1053,31 +1069,52 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
return ParserState.AwaitingNextPacket;
}
// Bounds the graceful drain. Send completions keep NextActivityCheck moving, so a slow peer
// could otherwise hold a closing socket open indefinitely.
internal void ArmDrainDeadline(long curTicks)
{
if (!_drainDeadlineArmed)
{
_drainDeadlineArmed = true;
_drainDeadline = curTicks + DrainTimeoutMs;
}
}
public void CheckAlive(long curTicks)
{
if (_socket == null || NextActivityCheck - curTicks >= 0)
if (_socket == null)
{
return;
}
if (_socket.DisconnectPending)
{
LogInfo("Force disconnecting stuck socket...");
_socketManager.DisconnectImmediate(_socket);
}
else
{
// Authenticated pre-game clients (login screens): send keep-alive instead of disconnecting.
// The 0xBD ClientVersionRequest resets NextActivityCheck via DataSent.
if (_account != null && Mobile == null)
ArmDrainDeadline(curTicks); // transport-initiated drains are first seen here
if (curTicks - _drainDeadline >= 0 || NextActivityCheck - curTicks < 0)
{
this.SendClientVersionRequest();
return;
LogInfo("Force disconnecting stuck socket...");
_socketManager.DisconnectImmediate(_socket);
}
LogInfo("Disconnecting due to inactivity...");
Disconnect("Disconnecting due to inactivity.");
return;
}
if (NextActivityCheck - curTicks >= 0)
{
return;
}
// Authenticated pre-game clients (login screens): send keep-alive instead of disconnecting.
// The 0xBD ClientVersionRequest resets NextActivityCheck via DataSent.
if (_account != null && Mobile == null)
{
this.SendClientVersionRequest();
return;
}
LogInfo("Disconnecting due to inactivity...");
Disconnect("Disconnecting due to inactivity.");
}
public void Trace(ReadOnlySpan<byte> buffer)
@ -1123,8 +1160,8 @@ public partial class NetState : IComparable<NetState>, IValueLinkListNode<NetSta
}
/// <summary>
/// Requests a graceful disconnect. The disconnect is queued and processed after the flush
/// queue in Slice(), ensuring Send() calls made in the same tick are processed first.
/// Requests a graceful disconnect. Processed after the flush queue in Slice(): sends made before
/// that handoff are flushed first, sends after it are dropped (see CannotSendPackets).
/// </summary>
public void Disconnect(string reason)
{

View file

@ -6,8 +6,8 @@ public static class OutgoingPackets
{
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static bool CannotSendPackets(this NetState ns) =>
// Do not check for NetState.Running. Packets are sent to a "disconnected" socket as part of the OnDisconnect events
// up until the socket is closed. Closing the connection is done synchronously, therefore packets will not be sent
// once the Mobile.NetState is null.
ns == null || ns.SocketHandle == 0 || ns.BlockAllPackets;
// Running is not checked: sends between Disconnect() and the Slice() handoff must still go out.
// After the handoff the socket is draining (DisconnectPending) or closing (!Connected); new writes
// only keep the buffer from draining.
ns == null || ns.SocketHandle == 0 || !ns._socket.Connected || ns._socket.DisconnectPending || ns.BlockAllPackets;
}

View file

@ -34,7 +34,7 @@
</Target>
<ItemGroup>
<ProjectReference Include="..\Logger\Logger.csproj" />
<PackageReference Include="IORingGroup" Version="1.0.10" />
<PackageReference Include="IORingGroup" Version="1.0.11" />
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
<PackageReference Include="LibDeflate.Bindings" Version="1.0.4" />
<PackageReference Include="System.IO.Hashing" Version="10.0.12" />