ModernUO/Projects/Server/Diagnostics/EventLoopProfiler.cs
Kamron Batman 0628902644
fix: harden idle-sleep scheduling against bad config and misattributed saves (#2567)
Follow-ups to #2559, from a review of the ported idle-sleep/scheduler-health changes.

### Fixes

- **`NetState.IsIdle` omitted `_pendingDisconnects`** — `Slice()` drains five queues; the property checked four. The other deferred work (`_connectingQueue`, alive checks, movement throttle) is time-gated and correctly excluded; the disconnect queue was the only ready-work omission. Impact was bounded (≤ one idle wait of delay), but the property's contract is "sleeping cannot strand pending work".
- **Neither new setting was clamped** (`Main.cs`):
  - `server.lateWakeThreshold: -1` made `late <= threshold` false for every sample even at zero late wakes, so from the second sample on, sleeping was re-suspended every second, forever — a permanent full-core spin whose only trace was a nonsense warning ("… at least 8ms late 0 time(s)").
  - `server.eventLoopIdleWaitMs: -1` disabled sleeping while the admin gump reported **Healthy** (it tested `== 0`).
  - Both now clamp to `>= 0` and log a warning naming the configured value. `-1` is a natural thing to reach for given the sibling key's doc says "set very high to disable".
- **World snapshots were misattributed to `StolenMs`** — `World.Snapshot` ran outside all five profiler phases, so a 3-second save inside a sample read as ~75% stolen, and `debugging-event-loop.md` teaches stolen = "the host ran something else". The diagnostic pointed operators at buying dedicated CPU for their own largest loop-thread stall. Saves now land in a new `WorldSnapshot` phase; `[LoopStats` iterates `PhaseCount` generically, so the report and CSV pick it up with no changes.
- **Admin gump conflated host-forced spin with configured spin** — when the startup probe finds no high-resolution wait support it zeroes the idle wait, after which the gump said "Spinning (configured)" and the operator's config said 2. New `Core.IdleSleepUnsupported` property; the gump now shows "Spinning - host cannot honor short waits" as a distinct fourth verdict. A genuinely configured 0 still reads "configured" (the probe only runs when the configured value was > 0).
- **The backoff-ceiling `Error` logged once per process lifetime** — `_loggedBackoffCeiling` never reset, and at the ceiling the method returns before the `Warning`, so a host that recovered (>60s clean streak) and later degraded back to the ceiling never re-logged the one operator-actionable message. The flag now resets with the clean-streak escalation reset.
- **Removed the unreachable "already suspended, extend" branch** — no sleeps occur while suspended, so `_lateWakes` stays 0 and every suspended sample early-returns before reaching it; with the threshold clamped it can never fire. If sleep gating ever changes, the normal path handles the case by counting a fresh episode.

`dev-docs/debugging-event-loop.md` updated to match (phase list + gump verdict table).

### Verification

- `dotnet build` clean (0 warnings) both normally and with `-p:EventLoopProfiling=true` (the snapshot phase only becomes live IL under the profiling flag).
2026-08-09 22:05:18 -07:00

234 lines
7.8 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: EventLoopProfiler.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;
namespace Server;
public enum LoopPhase
{
MobileDeltas,
ItemDeltas,
TimerSlice,
NetworkSlice,
LoopTasks,
WorldSnapshot,
}
/// <summary>
/// Event-loop time accounting, compiled out of normal builds. Build with
/// <c>-p:EventLoopProfiling=true</c> to enable; every hook is
/// <c>[Conditional("EVENT_LOOP_PROFILING")]</c>, so without the flag the call sites do not exist
/// in the IL and this class is dormant. See dev-docs/debugging-event-loop.md for how to read it.
/// </summary>
/// <remarks>
/// Each one-second sample decomposes wall time into work (per <see cref="LoopPhase"/>), sleep,
/// GC pause, and a stolen residual (wall - work - sleep): time the host ran something else.
/// Samples land in a ring buffer (~15 minutes) so a lag episode can be compared against the good
/// minutes on the same box, build, and world — the baseline RunUO's profiler never had.
/// </remarks>
public static class EventLoopProfiler
{
public const int PhaseCount = 6;
private const int RingSize = 900;
private const long SampleIntervalMs = 1000;
public struct Sample
{
public long WallStart; // Core.TickCount at sample start
public long WallMs; // sample length
public long Iterations;
public long Sleeps;
public double SleepMs; // total time blocked in WaitForCompletion
public double SleepOvershootMaxMs; // worst (elapsed - requested) this sample
public long LateWakes; // overshoot >= Timer.TickRate
public long WheelLagMaxMs; // worst wheel lateness observed at Slice entry
public long WakesIssued;
public long WakesElided;
public double GcPauseMs; // GC.GetTotalPauseDuration delta
public int Gen0;
public int Gen1;
public int Gen2;
public PhaseTimes Phases;
// Work the phases did not account for and the loop did not spend sleeping: host
// scheduling steals, and anything between the bracketed phases. GC pauses inside a
// phase or sleep inflate those measurements instead, so GcPauseMs overlaps rather
// than subtracts.
public double StolenMs
{
get
{
var known = SleepMs + Phases.Total;
return WallMs > known ? WallMs - known : 0;
}
}
}
[InlineArray(PhaseCount)]
public struct PhaseTimes
{
private double _element0;
public double Total
{
get
{
double total = 0;
for (var i = 0; i < PhaseCount; i++)
{
total += this[i];
}
return total;
}
}
}
private static readonly double _msPerTick = 1000.0 / Stopwatch.Frequency;
private static Sample[] _ring;
private static int _ringCount;
private static int _ringHead;
private static Sample _current;
private static long _phaseStartTimestamp;
private static long _sampleStartedAt;
private static TimeSpan _lastGcPause;
private static int _lastGen0;
private static int _lastGen1;
private static int _lastGen2;
/// <summary>Number of samples recorded so far (capped at the ring size).</summary>
public static int SampleCount => _ringCount;
/// <summary>The sample currently being accumulated (not yet in the ring).</summary>
public static Sample Current => _current;
/// <summary>
/// Copies the newest <paramref name="count"/> completed samples, oldest first.
/// </summary>
public static Sample[] History(int count = RingSize)
{
count = Math.Min(count, _ringCount);
var result = new Sample[count];
for (var i = 0; i < count; i++)
{
result[i] = _ring[(_ringHead - count + i + RingSize) % RingSize];
}
return result;
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void IterationStart(long tickCount)
{
if (_ring == null)
{
_ring = new Sample[RingSize];
_sampleStartedAt = tickCount;
_current.WallStart = tickCount;
_lastGcPause = GC.GetTotalPauseDuration();
_lastGen0 = GC.CollectionCount(0);
_lastGen1 = GC.CollectionCount(1);
_lastGen2 = GC.CollectionCount(2);
}
_current.Iterations++;
if (tickCount - _sampleStartedAt < SampleIntervalMs)
{
return;
}
_current.WallMs = tickCount - _sampleStartedAt;
var pause = GC.GetTotalPauseDuration();
_current.GcPauseMs = (pause - _lastGcPause).TotalMilliseconds;
_lastGcPause = pause;
var gen0 = GC.CollectionCount(0);
var gen1 = GC.CollectionCount(1);
var gen2 = GC.CollectionCount(2);
_current.Gen0 = gen0 - _lastGen0;
_current.Gen1 = gen1 - _lastGen1;
_current.Gen2 = gen2 - _lastGen2;
_lastGen0 = gen0;
_lastGen1 = gen1;
_lastGen2 = gen2;
_ring[_ringHead] = _current;
_ringHead = (_ringHead + 1) % RingSize;
if (_ringCount < RingSize)
{
_ringCount++;
}
_sampleStartedAt = tickCount;
_current = default;
_current.WallStart = tickCount;
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void PhaseStart(LoopPhase phase) => _phaseStartTimestamp = Stopwatch.GetTimestamp();
[Conditional("EVENT_LOOP_PROFILING")]
public static void PhaseEnd(LoopPhase phase) =>
_current.Phases[(int)phase] += (Stopwatch.GetTimestamp() - _phaseStartTimestamp) * _msPerTick;
[Conditional("EVENT_LOOP_PROFILING")]
public static void SleepEnd(int requestedMs, long elapsedMs)
{
_current.Sleeps++;
_current.SleepMs += elapsedMs;
var overshoot = elapsedMs - requestedMs;
if (overshoot > _current.SleepOvershootMaxMs)
{
_current.SleepOvershootMaxMs = overshoot;
}
if (overshoot >= Timer.TickRate)
{
_current.LateWakes++;
}
}
[Conditional("EVENT_LOOP_PROFILING")]
public static void WheelSlice(long deltaSinceTurn)
{
var lag = deltaSinceTurn - Timer.TickRate;
if (lag > _current.WheelLagMaxMs)
{
_current.WheelLagMaxMs = lag;
}
}
// Cross-thread; approximate counts are fine for diagnosis, so no interlocked.
[Conditional("EVENT_LOOP_PROFILING")]
public static void WakeSignal(bool elided)
{
if (elided)
{
_current.WakesElided++;
}
else
{
_current.WakesIssued++;
}
}
}