perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559)

## Problem

`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.

## Result

Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:

| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |

**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.

## How

The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.

**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.

**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.

## Configuration

| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |

## Diagnostics (compiled out by default)

`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.

## Verification

- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.

## Notes

The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.

Projects/Server.Tests/Tests/Network/EventLoopIdleTests.cs

@@ -0,0 +1,69 @@
+using Xunit;
+
+namespace Server.Tests;
+
+/// <summary>
+/// The event loop only sleeps when every queue it drains is empty. These drains are deliberately
+/// bounded -- ExecuteTasks stops at its per-frame cap -- so leftover work is normal and must keep
+/// the loop awake. Getting this wrong strands queued work for the length of a sleep.
+/// </summary>
+[Collection("Sequential Server Tests")]
+public class EventLoopIdleTests
+{
+    [Fact]
+    public void FreshContextIsEmpty()
+    {
+        var context = new EventLoopContext();
+
+        Assert.True(context.IsEmpty);
+    }
+
+    [Fact]
+    public void PostedWorkMakesContextNonEmpty()
+    {
+        var context = new EventLoopContext();
+
+        context.Post(() => { });
+
+        Assert.False(context.IsEmpty);
+    }
+
+    [Fact]
+    public void PriorityWorkMakesContextNonEmpty()
+    {
+        var context = new EventLoopContext();
+
+        context.Post(() => { }, EventLoopContext.Priority.High);
+
+        Assert.False(context.IsEmpty);
+    }
+
+    [Fact]
+    public void ContextIsEmptyAgainOnceDrained()
+    {
+        var context = new EventLoopContext();
+        context.Post(() => { });
+
+        context.ExecuteTasks();
+
+        Assert.True(context.IsEmpty);
+    }
+
+    [Fact]
+    public void WorkBeyondThePerFrameCapKeepsContextNonEmpty()
+    {
+        // The cap is what makes IsEmpty necessary: a single ExecuteTasks pass cannot be assumed
+        // to have drained everything, so the loop must not treat "I just ran tasks" as "idle".
+        const int perFrameCap = 128;
+        var context = new EventLoopContext(perFrameCap);
+
+        for (var i = 0; i < perFrameCap + 10; i++)
+        {
+            context.Post(() => { });
+        }
+
+        context.ExecuteTasks();
+
+        Assert.False(context.IsEmpty);
+    }
+}

Projects/Server/Diagnostics/EventLoopProfiler.cs

@@ -0,0 +1,233 @@
+/*************************************************************************
+ * ModernUO                                                              *
+ * Copyright 2019-2026 - ModernUO Development Team                       *
+ * Email: hi@modernuo.com                                                *
+ * File: EventLoopProfiler.cs                                            *
+ *                                                                       *
+ * This program is free software: you can redistribute it and/or modify  *
+ * it under the terms of the GNU General Public License as published by  *
+ * the Free Software Foundation, either version 3 of the License, or     *
+ * (at your option) any later version.                                   *
+ *                                                                       *
+ * You should have received a copy of the GNU General Public License     *
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>. *
+ *************************************************************************/
+
+using System;
+using System.Diagnostics;
+using System.Runtime.CompilerServices;
+
+namespace Server;
+
+public enum LoopPhase
+{
+    MobileDeltas,
+    ItemDeltas,
+    TimerSlice,
+    NetworkSlice,
+    LoopTasks,
+}
+
+/// <summary>
+/// Event-loop time accounting, compiled out of normal builds. Build with
+/// <c>-p:EventLoopProfiling=true</c> to enable; every hook is
+/// <c>[Conditional("EVENT_LOOP_PROFILING")]</c>, so without the flag the call sites do not exist
+/// in the IL and this class is dormant. See dev-docs/debugging-event-loop.md for how to read it.
+/// </summary>
+/// <remarks>
+/// Each one-second sample decomposes wall time into work (per <see cref="LoopPhase"/>), sleep,
+/// GC pause, and a stolen residual (wall - work - sleep): time the host ran something else.
+/// Samples land in a ring buffer (~15 minutes) so a lag episode can be compared against the good
+/// minutes on the same box, build, and world — the baseline RunUO's profiler never had.
+/// </remarks>
+public static class EventLoopProfiler
+{
+    public const int PhaseCount = 5;
+    private const int RingSize = 900;
+    private const long SampleIntervalMs = 1000;
+
+    public struct Sample
+    {
+        public long WallStart;               // Core.TickCount at sample start
+        public long WallMs;                  // sample length
+        public long Iterations;
+        public long Sleeps;
+        public double SleepMs;               // total time blocked in WaitForCompletion
+        public double SleepOvershootMaxMs;   // worst (elapsed - requested) this sample
+        public long LateWakes;               // overshoot >= Timer.TickRate
+        public long WheelLagMaxMs;           // worst wheel lateness observed at Slice entry
+        public long WakesIssued;
+        public long WakesElided;
+        public double GcPauseMs;             // GC.GetTotalPauseDuration delta
+        public int Gen0;
+        public int Gen1;
+        public int Gen2;
+        public PhaseTimes Phases;
+
+        // Work the phases did not account for and the loop did not spend sleeping: host
+        // scheduling steals, and anything between the bracketed phases. GC pauses inside a
+        // phase or sleep inflate those measurements instead, so GcPauseMs overlaps rather
+        // than subtracts.
+        public double StolenMs
+        {
+            get
+            {
+                var known = SleepMs + Phases.Total;
+                return WallMs > known ? WallMs - known : 0;
+            }
+        }
+    }
+
+    [InlineArray(PhaseCount)]
+    public struct PhaseTimes
+    {
+        private double _element0;
+
+        public double Total
+        {
+            get
+            {
+                double total = 0;
+                for (var i = 0; i < PhaseCount; i++)
+                {
+                    total += this[i];
+                }
+
+                return total;
+            }
+        }
+    }
+
+    private static readonly double _msPerTick = 1000.0 / Stopwatch.Frequency;
+
+    private static Sample[] _ring;
+    private static int _ringCount;
+    private static int _ringHead;
+
+    private static Sample _current;
+    private static long _phaseStartTimestamp;
+    private static long _sampleStartedAt;
+    private static TimeSpan _lastGcPause;
+    private static int _lastGen0;
+    private static int _lastGen1;
+    private static int _lastGen2;
+
+    /// <summary>Number of samples recorded so far (capped at the ring size).</summary>
+    public static int SampleCount => _ringCount;
+
+    /// <summary>The sample currently being accumulated (not yet in the ring).</summary>
+    public static Sample Current => _current;
+
+    /// <summary>
+    /// Copies the newest <paramref name="count"/> completed samples, oldest first.
+    /// </summary>
+    public static Sample[] History(int count = RingSize)
+    {
+        count = Math.Min(count, _ringCount);
+        var result = new Sample[count];
+        for (var i = 0; i < count; i++)
+        {
+            result[i] = _ring[(_ringHead - count + i + RingSize) % RingSize];
+        }
+
+        return result;
+    }
+
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void IterationStart(long tickCount)
+    {
+        if (_ring == null)
+        {
+            _ring = new Sample[RingSize];
+            _sampleStartedAt = tickCount;
+            _current.WallStart = tickCount;
+            _lastGcPause = GC.GetTotalPauseDuration();
+            _lastGen0 = GC.CollectionCount(0);
+            _lastGen1 = GC.CollectionCount(1);
+            _lastGen2 = GC.CollectionCount(2);
+        }
+
+        _current.Iterations++;
+
+        if (tickCount - _sampleStartedAt < SampleIntervalMs)
+        {
+            return;
+        }
+
+        _current.WallMs = tickCount - _sampleStartedAt;
+
+        var pause = GC.GetTotalPauseDuration();
+        _current.GcPauseMs = (pause - _lastGcPause).TotalMilliseconds;
+        _lastGcPause = pause;
+
+        var gen0 = GC.CollectionCount(0);
+        var gen1 = GC.CollectionCount(1);
+        var gen2 = GC.CollectionCount(2);
+        _current.Gen0 = gen0 - _lastGen0;
+        _current.Gen1 = gen1 - _lastGen1;
+        _current.Gen2 = gen2 - _lastGen2;
+        _lastGen0 = gen0;
+        _lastGen1 = gen1;
+        _lastGen2 = gen2;
+
+        _ring[_ringHead] = _current;
+        _ringHead = (_ringHead + 1) % RingSize;
+        if (_ringCount < RingSize)
+        {
+            _ringCount++;
+        }
+
+        _sampleStartedAt = tickCount;
+        _current = default;
+        _current.WallStart = tickCount;
+    }
+
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void PhaseStart(LoopPhase phase) => _phaseStartTimestamp = Stopwatch.GetTimestamp();
+
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void PhaseEnd(LoopPhase phase) =>
+        _current.Phases[(int)phase] += (Stopwatch.GetTimestamp() - _phaseStartTimestamp) * _msPerTick;
+
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void SleepEnd(int requestedMs, long elapsedMs)
+    {
+        _current.Sleeps++;
+        _current.SleepMs += elapsedMs;
+
+        var overshoot = elapsedMs - requestedMs;
+        if (overshoot > _current.SleepOvershootMaxMs)
+        {
+            _current.SleepOvershootMaxMs = overshoot;
+        }
+
+        if (overshoot >= Timer.TickRate)
+        {
+            _current.LateWakes++;
+        }
+    }
+
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void WheelSlice(long deltaSinceTurn)
+    {
+        var lag = deltaSinceTurn - Timer.TickRate;
+        if (lag > _current.WheelLagMaxMs)
+        {
+            _current.WheelLagMaxMs = lag;
+        }
+    }
+
+    // Cross-thread; approximate counts are fine for diagnosis, so no interlocked.
+    [Conditional("EVENT_LOOP_PROFILING")]
+    public static void WakeSignal(bool elided)
+    {
+        if (elided)
+        {
+            _current.WakesElided++;
+        }
+        else
+        {
+            _current.WakesIssued++;
+        }
+    }
+}

Projects/Server/EventLoopTasks.cs

@@ -42,10 +42,47 @@ public sealed class EventLoopContext : SynchronizationContext
 
     public override SynchronizationContext CreateCopy() => new EventLoopContext();
 
-    public void Post(Action d, Priority priority = Priority.Normal) =>
-        (priority == Priority.High ? _priorityQueue : _queue).Enqueue(d);
+    /// <summary>
+    /// True when no callbacks are waiting to run.
+    /// </summary>
+    /// <remarks>
+    /// <see cref="ExecuteTasks"/> drains at most <c>_maxPerFrame</c> callbacks, so work can
+    /// legitimately be left over. The event loop checks this before sleeping so a backlog keeps
+    /// it running instead.
+    /// </remarks>
+    public bool IsEmpty => _queue.IsEmpty && _priorityQueue.IsEmpty;
 
-    public override void Post(SendOrPostCallback d, object state) => _queue.Enqueue(() => d(state));
+    public void Post(Action d, Priority priority = Priority.Normal)
+    {
+        (priority == Priority.High ? _priorityQueue : _queue).Enqueue(d);
+        WakeEventLoop();
+    }
+
+    public override void Post(SendOrPostCallback d, object state)
+    {
+        _queue.Enqueue(() => d(state));
+        WakeEventLoop();
+    }
+
+    /// <summary>
+    /// Nudges the game loop in case it is asleep: the loop blocks on network I/O, which a queue
+    /// push alone does not signal.
+    /// </summary>
+    private void WakeEventLoop()
+    {
+        // A post from the loop thread cannot need a wake -- the loop is executing this very call
+        // -- and the signal is a syscall on every backend.
+        if (Thread.CurrentThread == _mainThread)
+        {
+            EventLoopProfiler.WakeSignal(elided: true);
+            return;
+        }
+
+        EventLoopProfiler.WakeSignal(elided: false);
+
+        // Safe before networking is configured and after teardown; NetState.Wake does nothing.
+        Network.NetState.Wake();
+    }
 
     public override void Send(SendOrPostCallback d, object state)
     {
@@ -63,6 +100,8 @@ public sealed class EventLoopContext : SynchronizationContext
             evt.Set();
         });
 
+        WakeEventLoop();
+
         evt.WaitOne();
     }
 

Projects/Server/Items/Item.cs

@@ -3325,6 +3325,12 @@ public partial class Item : IHued, IComparable<Item>, ISpawnable, IObjectPropert
         m_DeltaFlags &= ~flags;
     }
 
+    /// <summary>
+    /// True when deltas remain queued after a <see cref="ProcessDeltaQueue"/> pass, which is
+    /// bounded by the count it saw on entry. The event loop consults this before sleeping.
+    /// </summary>
+    public static bool HasQueuedDeltas => m_DeltaQueue.Count > 0;
+
     public static void ProcessDeltaQueue()
     {
         var limit = m_DeltaQueue.Count;

Projects/Server/Main.cs

@@ -39,10 +39,142 @@ public static class Core
 {
     private static readonly ILogger logger = LogFactory.GetLogger(typeof(Core));
 
-    private static bool _performProcessKill;
+    // Written from other threads (Kill, RequestSnapshot) and read by the event loop. Volatile
+    // because the loop now genuinely blocks between reads rather than spinning past them.
+    private static volatile bool _performProcessKill;
     private static bool _restartOnKill;
-    private static bool _performSnapshot;
+    private static volatile bool _performSnapshot;
     private static string _snapshotPath;
+
+    // A backstop, not a latency control: the wheel's tick rate bounds the sleep, so this only
+    // limits the damage if a wake signal is ever missed. Measured across 1/2/4/8ms, 2 is optimal.
+    private static int _eventLoopIdleWaitMs = 2;
+
+    /// <summary>
+    /// Longest the loop will block while idle, in milliseconds. 0 disables idle sleeping,
+    /// leaving the loop to spin; the adaptive backoff does the same thing temporarily when the
+    /// host keeps returning waits late.
+    /// </summary>
+    public static int EventLoopIdleWaitMs => _eventLoopIdleWaitMs;
+
+    /// <summary>
+    /// Whether idle sleeping is currently suspended because the host returned waits late.
+    /// </summary>
+    /// <remarks>
+    /// Compared by subtraction, never directly: tick counts can start enormous and wrap.
+    /// See dev-docs/tick-counts.md.
+    /// </remarks>
+    public static bool IdleSleepSuspended => _tickCount - _idleSleepSuspendedUntil < 0;
+
+    private const long HealthSampleIntervalMs = 1000;
+
+    // Backoff escalates by doubling: a fixed suspension oscillates forever on a persistently bad
+    // host, while doubling converges on "stop sleeping" within minutes yet still recovers from a
+    // transient problem.
+    private const long BackoffBaseMs = 5000;
+    private const long BackoffMaxMs = 120_000;
+    private const int BackoffMaxShift = 5;
+
+    // Clean streak that clears the escalation.
+    private const long BackoffResetAfterCleanMs = 60_000;
+
+    // A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can
+    // never miss a deadline; the only way sleeping harms the wheel is the wait returning late
+    // (the host descheduled the process). That overshoot is measured per sleep, which is why
+    // server work -- saves, heavy commands, deep timer callbacks -- cannot trip this backoff.
+    // Loop-thread only, so plain increments are safe.
+    private static int _lateWakes;
+
+    private static long _nextHealthSample;
+    private static long _idleSleepSuspendedUntil;
+    private static int _lateWakeThreshold = 1;
+    private static long _idleSleepBackoffs;
+    private static int _consecutiveBadSamples;
+    private static int _consecutiveBackoffs;
+    private static long _currentBackoffMs = BackoffBaseMs;
+    private static long _lastBackoffAt;
+    private static bool _loggedBackoffCeiling;
+
+    /// <summary>
+    /// Once a second, suspends idle sleeping (with escalating duration) if the host keeps
+    /// returning idle waits a full tick or more late.
+    /// </summary>
+    private static void CheckSchedulerHealth()
+    {
+        if (_tickCount - _nextHealthSample < 0)
+        {
+            return;
+        }
+
+        _nextHealthSample = _tickCount + HealthSampleIntervalMs;
+
+        var late = _lateWakes;
+        _lateWakes = 0;
+
+        if (late <= _lateWakeThreshold)
+        {
+            _consecutiveBadSamples = 0;
+            return;
+        }
+
+        // Require the condition to persist: any host can drop one sample to unrelated load, and a
+        // host that is genuinely oversubscribed stays that way, so it trips on the second sample.
+        if (++_consecutiveBadSamples < 2)
+        {
+            return;
+        }
+
+        if (_eventLoopIdleWaitMs <= 0)
+        {
+            return;
+        }
+
+        // Already suspended: extend rather than counting a fresh backoff episode.
+        if (_tickCount - _idleSleepSuspendedUntil < 0)
+        {
+            _idleSleepSuspendedUntil = _tickCount + _currentBackoffMs;
+            return;
+        }
+
+        // A long clean streak resets the escalation. Gated on the count rather than a
+        // "_lastBackoffAt > 0" sentinel because tick counts are not guaranteed positive.
+        if (_consecutiveBackoffs > 0 && _tickCount - _lastBackoffAt > BackoffResetAfterCleanMs)
+        {
+            _consecutiveBackoffs = 0;
+        }
+
+        _currentBackoffMs = Math.Min(BackoffBaseMs << Math.Min(_consecutiveBackoffs, BackoffMaxShift), BackoffMaxMs);
+        _consecutiveBackoffs++;
+        _lastBackoffAt = _tickCount;
+        _idleSleepSuspendedUntil = _tickCount + _currentBackoffMs;
+        _idleSleepBackoffs++;
+
+        if (_currentBackoffMs >= BackoffMaxMs)
+        {
+            // Escalation has run out of room; say so once in terms the operator can act on.
+            if (!_loggedBackoffCeiling)
+            {
+                _loggedBackoffCeiling = true;
+                logger.Error(
+                    "This host keeps returning idle waits late and sleeping has backed off {Count} times. " +
+                    "The process is not being scheduled promptly, which is typical of shared or burstable vCPUs. " +
+                    "Set server.eventLoopIdleWaitMs to 0 to disable sleeping permanently and trade a full core for latency.",
+                    _idleSleepBackoffs
+                );
+            }
+
+            return;
+        }
+
+        logger.Warning(
+            "This host returned a {Requested}ms idle wait at least {TickRate}ms late {Count} time(s) in the last " +
+            "second; idle sleeping suspended for {Duration}ms",
+            _eventLoopIdleWaitMs,
+            Timer.TickRate,
+            late,
+            _currentBackoffMs
+        );
+    }
     private static bool _crashed;
     private static string _baseDirectory;
 
@@ -111,14 +243,6 @@ public static class Core
 
     public static long Uptime => TickCount - _firstTick;
 
-    private static double _currentCPS;
-    private static double _averageCPS;
-    private static bool _cpsInitialized;
-
-    public static double CyclesPerSecond => _currentCPS;
-
-    public static double AverageCPS => _averageCPS;
-
     public static string BaseDirectory
     {
         get
@@ -235,6 +359,10 @@ public static class Core
     {
         _restartOnKill = restart;
         _performProcessKill = true;
+
+        // Callers are usually off-loop (console input, signal handlers). Without this the loop
+        // would not notice the request until it woke for some other reason.
+        NetState.Wake();
     }
 
     public static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
@@ -424,6 +552,13 @@ public static class Core
 
         ServerConfiguration.Load();
 
+        // 0 disables idle sleeping entirely (full-core spin, zero scheduling overhead).
+        _eventLoopIdleWaitMs = ServerConfiguration.GetSetting("server.eventLoopIdleWaitMs", 2);
+
+        // 16ms-budget misses per second before idle sleeping backs off. Raise to tolerate a
+        // jittery host; set very high to disable the backoff.
+        _lateWakeThreshold = ServerConfiguration.GetSetting("server.lateWakeThreshold", 1);
+
         var assemblyPath = Path.Join(BaseDirectory, AssembliesConfiguration);
 
         // Load UOContent.dll
@@ -453,6 +588,12 @@ public static class Core
         _now = DateTime.UtcNow;
         _firstTick = _tickCount = GetTimestamp();
 
+        // Seed schedule state from the first real tick: tick counts are not guaranteed to start
+        // anywhere near zero (hypervisor pass-through counters), so zero-initialized deadlines
+        // would compare wrong. See dev-docs/tick-counts.md.
+        _nextHealthSample = _tickCount + HealthSampleIntervalMs;
+        _idleSleepSuspendedUntil = _tickCount;
+
         Timer.Init(_tickCount);
 
         AssemblyHandler.Invoke("Configure");
@@ -469,34 +610,63 @@ public static class Core
         NetState.Start();
         PingServer.Start();
         EventSink.InvokeServerStarted();
+
+        // Without a high-resolution wait a 2ms request quantises to 15.625ms and the loop would
+        // quietly run a tick behind; spinning is the lesser evil and must not be silent. Only
+        // fires when both the ring's high-res timer and its timeBeginPeriod fallback failed.
+        if (_eventLoopIdleWaitMs > 0 && NetState.Ring?.SupportsHighResolutionWait == false)
+        {
+            logger.Error(
+                "This host cannot honor short waits (no high-resolution timer, and raising the system timer " +
+                "resolution failed). Idle sleeping is disabled. The loop will spin instead, using a full core."
+            );
+
+            _eventLoopIdleWaitMs = 0;
+        }
+
         RunEventLoop();
     }
 
+    /// <summary>
+    /// True when every queue the loop drains is empty, so sleeping cannot strand pending work.
+    /// The drains are bounded (ProcessDeltaQueue stops at the count seen on entry, ExecuteTasks
+    /// at its per-frame cap), so leftovers are normal and must keep the loop awake.
+    /// </summary>
+    private static bool IsIdle() =>
+        !Mobile.HasQueuedDeltas && !Item.HasQueuedDeltas && LoopContext.IsEmpty && NetState.IsIdle;
+
     public static void RunEventLoop()
     {
         try
         {
-            var lastRaw = Stopwatch.GetTimestamp();
-            const int interval = 100;
-            double frequency = Stopwatch.Frequency * interval;
-            const double alpha = 2.0 / 129; // EMA smoothing (≈128-sample window)
-
-            var sample = 0;
-
             while (!Closing)
             {
                 _tickCount = GetTimestamp();
                 _now = DateTime.UtcNow;
 
+                EventLoopProfiler.IterationStart(_tickCount);
+
+                EventLoopProfiler.PhaseStart(LoopPhase.MobileDeltas);
                 Mobile.ProcessDeltaQueue();
+                EventLoopProfiler.PhaseEnd(LoopPhase.MobileDeltas);
+
+                EventLoopProfiler.PhaseStart(LoopPhase.ItemDeltas);
                 Item.ProcessDeltaQueue();
+                EventLoopProfiler.PhaseEnd(LoopPhase.ItemDeltas);
+
+                EventLoopProfiler.PhaseStart(LoopPhase.TimerSlice);
                 Timer.Slice(_tickCount);
+                EventLoopProfiler.PhaseEnd(LoopPhase.TimerSlice);
 
                 // Handle networking
+                EventLoopProfiler.PhaseStart(LoopPhase.NetworkSlice);
                 NetState.Slice();
+                EventLoopProfiler.PhaseEnd(LoopPhase.NetworkSlice);
 
                 // Execute captured post-await methods (like Timer.Pause)
+                EventLoopProfiler.PhaseStart(LoopPhase.LoopTasks);
                 LoopContext.ExecuteTasks();
+                EventLoopProfiler.PhaseEnd(LoopPhase.LoopTasks);
 
                 Timer.CheckTimerPool(); // Check for pool depletion so we can async refill it.
 
@@ -513,29 +683,28 @@ public static class Core
                     break;
                 }
 
-                if (sample++ == interval)
+                CheckSchedulerHealth();
+
+                if (_eventLoopIdleWaitMs > 0 && _tickCount - _idleSleepSuspendedUntil >= 0 && IsIdle())
                 {
-                    sample = 0;
-                    var nowRaw = Stopwatch.GetTimestamp();
-
-                    _currentCPS = frequency / (nowRaw - lastRaw);
-
-                    if (!_cpsInitialized)
+                    // Re-read the clock: the loop body consumed real time, and a stale timestamp
+                    // would overstate the time to the next tick and sleep straight past it.
+                    var start = GetTimestamp();
+                    var due = Timer.MillisecondsUntilNextTick(start);
+                    if (due > 0)
                     {
-                        _averageCPS = _currentCPS;
-                        _cpsInitialized = true;
-                    }
-                    else
-                    {
-                        _averageCPS += alpha * (_currentCPS - _averageCPS);
-                    }
+                        var requested = (int)Math.Min(due, _eventLoopIdleWaitMs);
+                        NetState.WaitForCompletion(requested);
 
-                    lastRaw = nowRaw;
+                        var elapsed = GetTimestamp() - start;
+                        EventLoopProfiler.SleepEnd(requested, elapsed);
 
-                    var sleepMs = (int)Timer.MillisecondsUntilNextTick(_tickCount);
-                    if (sleepMs >= 2)
-                    {
-                        NetState.WaitForCompletion(sleepMs - 1);
+                        // A sleep is bounded by the next wheel turn, so only a wait the host
+                        // returned late can cost the wheel a deadline.
+                        if (elapsed - requested >= Timer.TickRate)
+                        {
+                            _lateWakes++;
+                        }
                     }
                 }
             }
@@ -553,6 +722,10 @@ public static class Core
     {
         _snapshotPath = snapshotPath;
         _performSnapshot = true;
+
+        // Save requests arrive off-loop. Wake so the snapshot starts now rather than after the
+        // loop happens to surface for another reason.
+        NetState.Wake();
     }
 
     public static void VerifySerialization()

Projects/Server/Mobiles/Mobile.cs

@@ -7834,6 +7834,12 @@ public partial class Mobile : IHued, IComparable<Mobile>, ISpawnable, IObjectPro
         }
     }
 
+    /// <summary>
+    /// True when deltas remain queued after a <see cref="ProcessDeltaQueue"/> pass, which is
+    /// bounded by the count it saw on entry. The event loop consults this before sleeping.
+    /// </summary>
+    public static bool HasQueuedDeltas => m_DeltaQueue.Count > 0;
+
     public static void ProcessDeltaQueue()
     {
         var limit = m_DeltaQueue.Count;

Projects/Server/Network/NetState/NetState.Network.cs

@@ -71,6 +71,24 @@ public partial class NetState
         _socketManager?.WaitForCompletion(timeoutMs);
     }
 
+    /// <summary>
+    /// Wakes the game loop if it is blocked in <see cref="WaitForCompletion"/>. Safe from any
+    /// thread; a no-op before networking is configured or after teardown. The signal is sticky,
+    /// so a wake racing the loop's decision to sleep is not lost.
+    /// </summary>
+    public static void Wake()
+    {
+        _socketManager?.Ring?.Wake();
+    }
+
+    /// <summary>
+    /// True when no queued network work remains for the loop to drain. <see cref="Slice"/> defers
+    /// work in several places, so an empty completion queue alone is not enough.
+    /// </summary>
+    internal static bool IsIdle =>
+        _throttled.Count == 0 && _throttledPending.Count == 0 &&
+        _flushPending.Count == 0 && _disposed.Count == 0;
+
     /// <summary>
     /// Gets the listening addresses that the server is bound to.
     /// </summary>

Projects/Server/Server.csproj

@@ -34,7 +34,7 @@
     </Target>
     <ItemGroup>
         <ProjectReference Include="..\Logger\Logger.csproj" />
-        <PackageReference Include="IORingGroup" Version="1.0.9" />
+        <PackageReference Include="IORingGroup" Version="1.0.10" />
         <PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.2" />
         <PackageReference Include="LibDeflate.Bindings" Version="1.0.4" />
         <PackageReference Include="System.IO.Hashing" Version="10.0.10" />

Projects/Server/Timer/Timer.TimerWheel.cs

@@ -51,8 +51,15 @@ public partial class Timer
         }
     }
 
+    /// <summary>
+    /// Milliseconds of simulated time one wheel turn advances.
+    /// </summary>
+    public static int TickRate => _tickRate;
+
     public static void Slice(long tickCount)
     {
+        EventLoopProfiler.WheelSlice(tickCount - _lastTickTurned);
+
         var deltaSinceTurn = tickCount - _lastTickTurned;
         while (deltaSinceTurn >= _tickRate)
         {

Projects/UOContent/Commands/LoopStats.cs

@@ -0,0 +1,117 @@
+#if EVENT_LOOP_PROFILING
+using System;
+using System.Globalization;
+using System.IO;
+using Server.Logging;
+
+namespace Server.Commands;
+
+/// <summary>
+/// Reports the event-loop time decomposition recorded by <see cref="EventLoopProfiler"/>.
+/// Only compiled when the server is built with -p:EventLoopProfiling=true.
+/// See dev-docs/debugging-event-loop.md for how to read the output.
+/// </summary>
+public static class LoopStats
+{
+    private static readonly ILogger logger = LogFactory.GetLogger(typeof(LoopStats));
+
+    public static void Configure()
+    {
+        CommandSystem.Register("LoopStats", AccessLevel.Administrator, LoopStats_OnCommand);
+    }
+
+    [Usage("LoopStats")]
+    [Description("Summarizes the last minute of event-loop time accounting and writes the full history to a CSV.")]
+    private static void LoopStats_OnCommand(CommandEventArgs e)
+    {
+        var history = EventLoopProfiler.History();
+        if (history.Length == 0)
+        {
+            e.Mobile.SendMessage("No samples recorded yet.");
+            return;
+        }
+
+        var window = Math.Min(60, history.Length);
+
+        double wall = 0, sleep = 0, gc = 0, stolen = 0, stolenMax = 0;
+        long iterations = 0, sleeps = 0, lateWakes = 0, wheelLagMax = 0;
+        Span<double> phases = stackalloc double[EventLoopProfiler.PhaseCount];
+        Span<double> phaseMax = stackalloc double[EventLoopProfiler.PhaseCount];
+
+        for (var i = history.Length - window; i < history.Length; i++)
+        {
+            ref var s = ref history[i];
+            wall += s.WallMs;
+            sleep += s.SleepMs;
+            gc += s.GcPauseMs;
+            stolen += s.StolenMs;
+            iterations += s.Iterations;
+            sleeps += s.Sleeps;
+            lateWakes += s.LateWakes;
+
+            if (s.StolenMs > stolenMax)
+            {
+                stolenMax = s.StolenMs;
+            }
+
+            if (s.WheelLagMaxMs > wheelLagMax)
+            {
+                wheelLagMax = s.WheelLagMaxMs;
+            }
+
+            for (var p = 0; p < EventLoopProfiler.PhaseCount; p++)
+            {
+                phases[p] += s.Phases[p];
+                if (s.Phases[p] > phaseMax[p])
+                {
+                    phaseMax[p] = s.Phases[p];
+                }
+            }
+        }
+
+        e.Mobile.SendMessage($"Loop, last {window}s of wall time {wall:F0}ms:");
+        e.Mobile.SendMessage($"  sleep {100 * sleep / wall:F1}%, gc {100 * gc / wall:F1}%, stolen {100 * stolen / wall:F1}% (worst {stolenMax:F0}ms/s)");
+
+        for (var p = 0; p < EventLoopProfiler.PhaseCount; p++)
+        {
+            e.Mobile.SendMessage($"  {(LoopPhase)p}: {100 * phases[p] / wall:F1}% (worst {phaseMax[p]:F0}ms/s)");
+        }
+
+        e.Mobile.SendMessage($"  {iterations} iterations, {sleeps} sleeps, {lateWakes} late wakes, worst wheel lag {wheelLagMax}ms");
+
+        var path = Path.Combine(Core.BaseDirectory, $"loopstats-{Core.Now:yyyyMMdd-HHmmss}.csv");
+        WriteCsv(path, history);
+        e.Mobile.SendMessage($"Full history ({history.Length} samples) written to {path}");
+        logger.Information("Loop stats dumped to {Path}", path);
+    }
+
+    private static void WriteCsv(string path, EventLoopProfiler.Sample[] history)
+    {
+        using var writer = new StreamWriter(path);
+        writer.Write("wallStart,wallMs,iterations,sleeps,sleepMs,sleepOvershootMaxMs,lateWakes,wheelLagMaxMs,wakesIssued,wakesElided,gcPauseMs,gen0,gen1,gen2,stolenMs");
+        for (var p = 0; p < EventLoopProfiler.PhaseCount; p++)
+        {
+            writer.Write(',');
+            writer.Write((LoopPhase)p);
+        }
+
+        writer.WriteLine();
+
+        for (var i = 0; i < history.Length; i++)
+        {
+            ref var s = ref history[i];
+            writer.Write(string.Create(
+                CultureInfo.InvariantCulture,
+                $"{s.WallStart},{s.WallMs},{s.Iterations},{s.Sleeps},{s.SleepMs:F2},{s.SleepOvershootMaxMs:F2},{s.LateWakes},{s.WheelLagMaxMs},{s.WakesIssued},{s.WakesElided},{s.GcPauseMs:F2},{s.Gen0},{s.Gen1},{s.Gen2},{s.StolenMs:F2}"
+            ));
+            for (var p = 0; p < EventLoopProfiler.PhaseCount; p++)
+            {
+                writer.Write(',');
+                writer.Write(string.Create(CultureInfo.InvariantCulture, $"{s.Phases[p]:F2}"));
+            }
+
+            writer.WriteLine();
+        }
+    }
+}
+#endif

Projects/UOContent/Gumps/AdminGump.cs

@@ -227,9 +227,11 @@ namespace Server.Gumps
                     }
                 case AdminGumpPage.Information_Perf:
                     {
-                        AddLabel(20, 130, LabelHue, "Cycles Per Second:");
-                        AddLabel(40, 150, LabelHue, $"Current: {Core.CyclesPerSecond:N2}");
-                        AddLabel(40, 170, LabelHue, $"Average: {Core.AverageCPS:N2}");
+                        var loopStatus = Core.EventLoopIdleWaitMs == 0 ? "Spinning (configured)" :
+                            Core.IdleSleepSuspended ? "Sleep suspended - host returning waits late" : "Healthy";
+
+                        AddLabel(20, 130, LabelHue, "Event Loop:");
+                        AddLabel(40, 150, LabelHue, loopStatus);
 
                         using var sb = ValueStringBuilder.Create();