294dcd94a0
## Summary Two related fixes on the outbound path: 1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it. 2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state. ## 1. Send-path stall (RIO) RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued. Measured on a 50ms-RTT production shard: | | before | after | |---|---|---| | in-game latency, data flowing | **101–146 ms** | **48–51 ms** | | p95 | ~135 ms | 52.8 ms | | samples > 70 ms | 20 | **0** | The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did. Two things worth recording, because they explain why this went unnoticed: - As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones. - The same measurement at loopback RTT is **microseconds**, so local testing could never surface it. New settings, both restart-time: - **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`. - **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling. ## 2. Send buffer full `NetState.Send` had three failure modes once the buffer filled, none of them visible: | writable | behaviour | |---|---| | `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect | | `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way | | `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** | The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space. `Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`. ## Testing `NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical. Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8. ## Notes for reviewers - Upstream change: modernuo/IORingGroup#9. - The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise. - Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
55 lines
3.4 KiB
XML
55 lines
3.4 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current">
|
|
<PropertyGroup>
|
|
<Product>ModernUO Core</Product>
|
|
<OutDir>..\..\Distribution</OutDir>
|
|
<PublishDir>..\..\Distribution</PublishDir>
|
|
<Version>0.0.0</Version>
|
|
<Configurations>Debug;Release;Analyze</Configurations>
|
|
<RootNamespace>Server</RootNamespace>
|
|
<PackageId>Server</PackageId>
|
|
</PropertyGroup>
|
|
<Target Name="CleanPub" AfterTargets="Clean">
|
|
<Message Text="Server: Removing distribution files..." />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName)" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).deps.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).dll.config" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).exe" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).pdb" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).runtimeconfig.dev.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).runtimeconfig.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\ModernUO.Serialization.Annotations.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\CommunityToolkit.HighPerformance.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\IORingGroup.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\ioring.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\ref\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.Sinks.Async.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.Sinks.Console.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\LibDeflate.Bindings.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\libdeflate.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\libdeflate.dylib" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\System.IO.Hashing.dll" ContinueOnError="true" />
|
|
</Target>
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\Logger\Logger.csproj" />
|
|
<PackageReference Include="IORingGroup" Version="1.0.8" />
|
|
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
|
|
<PackageReference Include="LibDeflate.Bindings" Version="1.0.3" />
|
|
<PackageReference Include="System.IO.Hashing" Version="10.0.10" />
|
|
|
|
<PackageReference Include="ModernUO.Serialization.Annotations" Version="2.14.2" />
|
|
<PackageReference Include="ModernUO.Serialization.Generator" Version="2.14.3" />
|
|
<PackageReference Update="Serilog" Version="4.4.0" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<AdditionalFiles Include="Migrations/*.v*.json" />
|
|
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
|
<_Parameter1>Server.Tests</_Parameter1>
|
|
</AssemblyAttribute>
|
|
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
|
<_Parameter1>UOContent.Tests</_Parameter1>
|
|
</AssemblyAttribute>
|
|
</ItemGroup>
|
|
</Project>
|