warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559)

Browse source 
## Problem

`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.

## Result

Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:

| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |

**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.

## How

The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.

**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.

**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.

## Configuration

| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |

## Diagnostics (compiled out by default)

`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.

## Verification

- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.

## Notes

The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
This commit is contained in:
Kamron Batman 2026-08-09 13:24:59 -07:00 • committed by GitHub
parent a7e65aab01
commit 6d846b11e5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
19 changed files with 1066 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

233
Projects/Server/Diagnostics/EventLoopProfiler.cs Normal file
View file

@ -0,0 +1,233 @@
/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: EventLoopProfiler.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;
namespace Server;
public enum LoopPhase
{
MobileDeltas,
ItemDeltas,
TimerSlice,
NetworkSlice,
LoopTasks,
}
/// <summary>
/// Event-loop time accounting, compiled out of normal builds. Build with
/// <c>-p:EventLoopProfiling=true</c> to enable; every hook is
/// <c>[Conditional("EVENT_LOOP_PROFILING")]</c>, so without the flag the call sites do not exist
/// in the IL and this class is dormant. See dev-docs/debugging-event-loop.md for how to read it.
/// </summary>
/// <remarks>
/// Each one-second sample decomposes wall time into work (per <see cref="LoopPhase"/>), sleep,
/// GC pause, and a stolen residual (wall - work - sleep): time the host ran something else.
/// Samples land in a ring buffer (~15 minutes) so a lag episode can be compared against the good
/// minutes on the same box, build, and world — the baseline RunUO's profiler never had.
/// </remarks>
public static class EventLoopProfiler
{
public const int PhaseCount = 5;
private const int RingSize = 900;
private const long SampleIntervalMs = 1000;
public struct Sample
{
public long WallStart; // Core.TickCount at sample start
public long WallMs; // sample length
public long Iterations;
public long Sleeps;
public double SleepMs; // total time blocked in WaitForCompletion
public double SleepOvershootMaxMs; // worst (elapsed - requested) this sample
public long LateWakes; // overshoot >= Timer.TickRate
public long WheelLagMaxMs; // worst wheel lateness observed at Slice entry
public long WakesIssued;
public long WakesElided;
public double GcPauseMs; // GC.GetTotalPauseDuration delta
public int Gen0;
public int Gen1;
public int Gen2;
public PhaseTimes Phases;
// Work the phases did not account for and the loop did not spend sleeping: host
// scheduling steals, and anything between the bracketed phases. GC pauses inside a
// phase or sleep inflate those measurements instead, so GcPauseMs overlaps rather
// than subtracts.
public double StolenMs
{
get
{
var known = SleepMs + Phases.Total;
return WallMs > known ? WallMs - known : 0;
}
}
}
[InlineArray(PhaseCount)]
public struct PhaseTimes
{
private double _element0;
public double Total
{
get
{
double total = 0;
for (var i = 0; i < PhaseCount; i++)
{
total += this[i];
}
return total;
}
}
}
private static readonly double _msPerTick = 1000.0 / Stopwatch.Frequency;
private static Sample[] _ring;
private static int _ringCount;
private static int _ringHead;
private static Sample _current;
private static long _phaseStartTimestamp;
private static long _sampleStartedAt;
private static TimeSpan _lastGcPause;
private static int _lastGen0;
private static int _lastGen1;
private static int _lastGen2;
/// <summary>Number of samples recorded so far (capped at the ring size).</summary>
public static int SampleCount => _ringCount;
/// <summary>The sample currently being accumulated (not yet in the ring).</summary>
public static Sample Current => _current;
/// <summary>
/// Copies the newest <paramref name="count"/> completed samples, oldest first.
/// </summary>
public static Sample[] History(int count = RingSize)
{
count = Math.Min(count, _ringCount);
var result = new Sample[count];
for (var i = 0; i < count; i++)
{
result[i] = _ring[(_ringHead - count + i + RingSize) % RingSize];
}
return result;
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void IterationStart(long tickCount)
{
if (_ring == null)
{
_ring = new Sample[RingSize];
_sampleStartedAt = tickCount;
_current.WallStart = tickCount;
_lastGcPause = GC.GetTotalPauseDuration();
_lastGen0 = GC.CollectionCount(0);
_lastGen1 = GC.CollectionCount(1);
_lastGen2 = GC.CollectionCount(2);
}
_current.Iterations++;
if (tickCount - _sampleStartedAt < SampleIntervalMs)
{
return;
}
_current.WallMs = tickCount - _sampleStartedAt;
var pause = GC.GetTotalPauseDuration();
_current.GcPauseMs = (pause - _lastGcPause).TotalMilliseconds;
_lastGcPause = pause;
var gen0 = GC.CollectionCount(0);
var gen1 = GC.CollectionCount(1);
var gen2 = GC.CollectionCount(2);
_current.Gen0 = gen0 - _lastGen0;
_current.Gen1 = gen1 - _lastGen1;
_current.Gen2 = gen2 - _lastGen2;
_lastGen0 = gen0;
_lastGen1 = gen1;
_lastGen2 = gen2;
_ring[_ringHead] = _current;
_ringHead = (_ringHead + 1) % RingSize;
if (_ringCount < RingSize)
{
_ringCount++;
}
_sampleStartedAt = tickCount;
_current = default;
_current.WallStart = tickCount;
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void PhaseStart(LoopPhase phase) => _phaseStartTimestamp = Stopwatch.GetTimestamp();
[Conditional("EVENT_LOOP_PROFILING")]
public static void PhaseEnd(LoopPhase phase) =>
_current.Phases[(int)phase] += (Stopwatch.GetTimestamp() - _phaseStartTimestamp) * _msPerTick;
[Conditional("EVENT_LOOP_PROFILING")]
public static void SleepEnd(int requestedMs, long elapsedMs)
{
_current.Sleeps++;
_current.SleepMs += elapsedMs;
var overshoot = elapsedMs - requestedMs;
if (overshoot > _current.SleepOvershootMaxMs)
{
_current.SleepOvershootMaxMs = overshoot;
}
if (overshoot >= Timer.TickRate)
{
_current.LateWakes++;
}
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void WheelSlice(long deltaSinceTurn)
{
var lag = deltaSinceTurn - Timer.TickRate;
if (lag > _current.WheelLagMaxMs)
{
_current.WheelLagMaxMs = lag;
}
}
// Cross-thread; approximate counts are fine for diagnosis, so no interlocked.
[Conditional("EVENT_LOOP_PROFILING")]
public static void WakeSignal(bool elided)
{
if (elided)
{
_current.WakesElided++;
}
else
{
_current.WakesIssued++;
}
}
}