Follow-ups to #2559, from a review of the ported idle-sleep/scheduler-health changes. ### Fixes - **`NetState.IsIdle` omitted `_pendingDisconnects`** — `Slice()` drains five queues; the property checked four. The other deferred work (`_connectingQueue`, alive checks, movement throttle) is time-gated and correctly excluded; the disconnect queue was the only ready-work omission. Impact was bounded (≤ one idle wait of delay), but the property's contract is "sleeping cannot strand pending work". - **Neither new setting was clamped** (`Main.cs`): - `server.lateWakeThreshold: -1` made `late <= threshold` false for every sample even at zero late wakes, so from the second sample on, sleeping was re-suspended every second, forever — a permanent full-core spin whose only trace was a nonsense warning ("… at least 8ms late 0 time(s)"). - `server.eventLoopIdleWaitMs: -1` disabled sleeping while the admin gump reported **Healthy** (it tested `== 0`). - Both now clamp to `>= 0` and log a warning naming the configured value. `-1` is a natural thing to reach for given the sibling key's doc says "set very high to disable". - **World snapshots were misattributed to `StolenMs`** — `World.Snapshot` ran outside all five profiler phases, so a 3-second save inside a sample read as ~75% stolen, and `debugging-event-loop.md` teaches stolen = "the host ran something else". The diagnostic pointed operators at buying dedicated CPU for their own largest loop-thread stall. Saves now land in a new `WorldSnapshot` phase; `[LoopStats` iterates `PhaseCount` generically, so the report and CSV pick it up with no changes. - **Admin gump conflated host-forced spin with configured spin** — when the startup probe finds no high-resolution wait support it zeroes the idle wait, after which the gump said "Spinning (configured)" and the operator's config said 2. New `Core.IdleSleepUnsupported` property; the gump now shows "Spinning - host cannot honor short waits" as a distinct fourth verdict. A genuinely configured 0 still reads "configured" (the probe only runs when the configured value was > 0). - **The backoff-ceiling `Error` logged once per process lifetime** — `_loggedBackoffCeiling` never reset, and at the ceiling the method returns before the `Warning`, so a host that recovered (>60s clean streak) and later degraded back to the ceiling never re-logged the one operator-actionable message. The flag now resets with the clean-streak escalation reset. - **Removed the unreachable "already suspended, extend" branch** — no sleeps occur while suspended, so `_lateWakes` stays 0 and every suspended sample early-returns before reaching it; with the threshold clamped it can never fire. If sleep gating ever changes, the normal path handles the case by counting a fresh episode. `dev-docs/debugging-event-loop.md` updated to match (phase list + gump verdict table). ### Verification - `dotnet build` clean (0 warnings) both normally and with `-p:EventLoopProfiling=true` (the snapshot phase only becomes live IL under the profiling flag).
234 lines
7.8 KiB
C#
234 lines
7.8 KiB
C#
/*************************************************************************
|
|
* ModernUO *
|
|
* Copyright 2019-2026 - ModernUO Development Team *
|
|
* Email: hi@modernuo.com *
|
|
* File: EventLoopProfiler.cs *
|
|
* *
|
|
* This program is free software: you can redistribute it and/or modify *
|
|
* it under the terms of the GNU General Public License as published by *
|
|
* the Free Software Foundation, either version 3 of the License, or *
|
|
* (at your option) any later version. *
|
|
* *
|
|
* You should have received a copy of the GNU General Public License *
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
|
|
*************************************************************************/
|
|
|
|
using System;
|
|
using System.Diagnostics;
|
|
using System.Runtime.CompilerServices;
|
|
|
|
namespace Server;
|
|
|
|
public enum LoopPhase
|
|
{
|
|
MobileDeltas,
|
|
ItemDeltas,
|
|
TimerSlice,
|
|
NetworkSlice,
|
|
LoopTasks,
|
|
WorldSnapshot,
|
|
}
|
|
|
|
/// <summary>
|
|
/// Event-loop time accounting, compiled out of normal builds. Build with
|
|
/// <c>-p:EventLoopProfiling=true</c> to enable; every hook is
|
|
/// <c>[Conditional("EVENT_LOOP_PROFILING")]</c>, so without the flag the call sites do not exist
|
|
/// in the IL and this class is dormant. See dev-docs/debugging-event-loop.md for how to read it.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Each one-second sample decomposes wall time into work (per <see cref="LoopPhase"/>), sleep,
|
|
/// GC pause, and a stolen residual (wall - work - sleep): time the host ran something else.
|
|
/// Samples land in a ring buffer (~15 minutes) so a lag episode can be compared against the good
|
|
/// minutes on the same box, build, and world — the baseline RunUO's profiler never had.
|
|
/// </remarks>
|
|
public static class EventLoopProfiler
|
|
{
|
|
public const int PhaseCount = 6;
|
|
private const int RingSize = 900;
|
|
private const long SampleIntervalMs = 1000;
|
|
|
|
public struct Sample
|
|
{
|
|
public long WallStart; // Core.TickCount at sample start
|
|
public long WallMs; // sample length
|
|
public long Iterations;
|
|
public long Sleeps;
|
|
public double SleepMs; // total time blocked in WaitForCompletion
|
|
public double SleepOvershootMaxMs; // worst (elapsed - requested) this sample
|
|
public long LateWakes; // overshoot >= Timer.TickRate
|
|
public long WheelLagMaxMs; // worst wheel lateness observed at Slice entry
|
|
public long WakesIssued;
|
|
public long WakesElided;
|
|
public double GcPauseMs; // GC.GetTotalPauseDuration delta
|
|
public int Gen0;
|
|
public int Gen1;
|
|
public int Gen2;
|
|
public PhaseTimes Phases;
|
|
|
|
// Work the phases did not account for and the loop did not spend sleeping: host
|
|
// scheduling steals, and anything between the bracketed phases. GC pauses inside a
|
|
// phase or sleep inflate those measurements instead, so GcPauseMs overlaps rather
|
|
// than subtracts.
|
|
public double StolenMs
|
|
{
|
|
get
|
|
{
|
|
var known = SleepMs + Phases.Total;
|
|
return WallMs > known ? WallMs - known : 0;
|
|
}
|
|
}
|
|
}
|
|
|
|
[InlineArray(PhaseCount)]
|
|
public struct PhaseTimes
|
|
{
|
|
private double _element0;
|
|
|
|
public double Total
|
|
{
|
|
get
|
|
{
|
|
double total = 0;
|
|
for (var i = 0; i < PhaseCount; i++)
|
|
{
|
|
total += this[i];
|
|
}
|
|
|
|
return total;
|
|
}
|
|
}
|
|
}
|
|
|
|
private static readonly double _msPerTick = 1000.0 / Stopwatch.Frequency;
|
|
|
|
private static Sample[] _ring;
|
|
private static int _ringCount;
|
|
private static int _ringHead;
|
|
|
|
private static Sample _current;
|
|
private static long _phaseStartTimestamp;
|
|
private static long _sampleStartedAt;
|
|
private static TimeSpan _lastGcPause;
|
|
private static int _lastGen0;
|
|
private static int _lastGen1;
|
|
private static int _lastGen2;
|
|
|
|
/// <summary>Number of samples recorded so far (capped at the ring size).</summary>
|
|
public static int SampleCount => _ringCount;
|
|
|
|
/// <summary>The sample currently being accumulated (not yet in the ring).</summary>
|
|
public static Sample Current => _current;
|
|
|
|
/// <summary>
|
|
/// Copies the newest <paramref name="count"/> completed samples, oldest first.
|
|
/// </summary>
|
|
public static Sample[] History(int count = RingSize)
|
|
{
|
|
count = Math.Min(count, _ringCount);
|
|
var result = new Sample[count];
|
|
for (var i = 0; i < count; i++)
|
|
{
|
|
result[i] = _ring[(_ringHead - count + i + RingSize) % RingSize];
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void IterationStart(long tickCount)
|
|
{
|
|
if (_ring == null)
|
|
{
|
|
_ring = new Sample[RingSize];
|
|
_sampleStartedAt = tickCount;
|
|
_current.WallStart = tickCount;
|
|
_lastGcPause = GC.GetTotalPauseDuration();
|
|
_lastGen0 = GC.CollectionCount(0);
|
|
_lastGen1 = GC.CollectionCount(1);
|
|
_lastGen2 = GC.CollectionCount(2);
|
|
}
|
|
|
|
_current.Iterations++;
|
|
|
|
if (tickCount - _sampleStartedAt < SampleIntervalMs)
|
|
{
|
|
return;
|
|
}
|
|
|
|
_current.WallMs = tickCount - _sampleStartedAt;
|
|
|
|
var pause = GC.GetTotalPauseDuration();
|
|
_current.GcPauseMs = (pause - _lastGcPause).TotalMilliseconds;
|
|
_lastGcPause = pause;
|
|
|
|
var gen0 = GC.CollectionCount(0);
|
|
var gen1 = GC.CollectionCount(1);
|
|
var gen2 = GC.CollectionCount(2);
|
|
_current.Gen0 = gen0 - _lastGen0;
|
|
_current.Gen1 = gen1 - _lastGen1;
|
|
_current.Gen2 = gen2 - _lastGen2;
|
|
_lastGen0 = gen0;
|
|
_lastGen1 = gen1;
|
|
_lastGen2 = gen2;
|
|
|
|
_ring[_ringHead] = _current;
|
|
_ringHead = (_ringHead + 1) % RingSize;
|
|
if (_ringCount < RingSize)
|
|
{
|
|
_ringCount++;
|
|
}
|
|
|
|
_sampleStartedAt = tickCount;
|
|
_current = default;
|
|
_current.WallStart = tickCount;
|
|
}
|
|
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void PhaseStart(LoopPhase phase) => _phaseStartTimestamp = Stopwatch.GetTimestamp();
|
|
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void PhaseEnd(LoopPhase phase) =>
|
|
_current.Phases[(int)phase] += (Stopwatch.GetTimestamp() - _phaseStartTimestamp) * _msPerTick;
|
|
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void SleepEnd(int requestedMs, long elapsedMs)
|
|
{
|
|
_current.Sleeps++;
|
|
_current.SleepMs += elapsedMs;
|
|
|
|
var overshoot = elapsedMs - requestedMs;
|
|
if (overshoot > _current.SleepOvershootMaxMs)
|
|
{
|
|
_current.SleepOvershootMaxMs = overshoot;
|
|
}
|
|
|
|
if (overshoot >= Timer.TickRate)
|
|
{
|
|
_current.LateWakes++;
|
|
}
|
|
}
|
|
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void WheelSlice(long deltaSinceTurn)
|
|
{
|
|
var lag = deltaSinceTurn - Timer.TickRate;
|
|
if (lag > _current.WheelLagMaxMs)
|
|
{
|
|
_current.WheelLagMaxMs = lag;
|
|
}
|
|
}
|
|
|
|
// Cross-thread; approximate counts are fine for diagnosis, so no interlocked.
|
|
[Conditional("EVENT_LOOP_PROFILING")]
|
|
public static void WakeSignal(bool elided)
|
|
{
|
|
if (elided)
|
|
{
|
|
_current.WakesElided++;
|
|
}
|
|
else
|
|
{
|
|
_current.WakesIssued++;
|
|
}
|
|
}
|
|
}
|