ModernUO/Projects/Server
Kamron Batman 6d846b11e5
perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559)
## Problem

`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.

## Result

Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:

| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |

**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.

## How

The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.

**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.

**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.

## Configuration

| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |

## Diagnostics (compiled out by default)

`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.

## Verification

- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.

## Notes

The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
2026-08-09 13:24:59 -07:00
..
Buffers feat(buffers): add :L lowercase format spec to RawInterpolatedStringHandler (#2440) 2026-05-03 18:24:57 -07:00
Client feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Collections feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) 2026-07-25 11:59:37 -07:00
Comparers fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Compression feat(pathfinding): .swb format v8 compact index (#3b) (#2471) 2026-06-07 01:22:20 -07:00
Configuration fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
Console fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
ContextMenus fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Diagnostics perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
Events perf(login): run password hashing on a parked worker thread (#2566) 2026-08-09 00:13:34 -07:00
Exceptions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
GarbageCollection chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Geometry fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Items perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
Json fix(json): Rectangle3DConverter loses a z-level on write (#2506) 2026-06-25 23:42:49 -07:00
Localization feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Maps fix: Fixes pathfinding prebake and pathfinding multi-fallthrough. (#2478) 2026-06-08 11:59:24 -07:00
Menus feat: Pre-Publish 14 Crafting (supersedes #2181, #2381) (#2476) 2026-06-07 20:27:22 -07:00
Migrations refactor: decompose mobile/corpse hair, delete VirtualHairInfo, fix removal serial (#2462) (#2463) 2026-06-06 14:33:28 -07:00
Mobiles perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
Network perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
PropertyList fix(opl): refuse property list invalidation raised from inside GetProperties (#2555) 2026-07-28 21:29:03 -07:00
Random fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Regions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serialization perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Targeting chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Text feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
TileMatrix feat(pathfinding): JSONL recorder + public bake helpers (#2449) 2026-05-06 13:06:28 -07:00
Timer perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
TimeZones fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Utilities fix: Removes unnecessary dictionary removal guards (#2565) 2026-08-08 11:50:01 -07:00
World perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
AssemblyHandler.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Attributes.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
CityInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Commands.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Effects.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EntityFinalizationTracker.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EventLoopTasks.cs perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
ExpansionInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
FeatureFlags.cs fix: Streamlines insurance. Insurance only executes when enabled. (#2550) 2026-07-26 09:47:49 -07:00
Guild.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
HuePicker.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
IAccount.cs fix: Fixes looking up accounts that were renamed. (#2078) 2025-01-18 17:11:43 -08:00
IEntity.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Interfaces.cs feat: Add CanSpawnMobile overload with props Z-range support. (#2293) 2025-12-27 17:01:15 -08:00
IVirtualCheckGump.cs feat: Moves gumps out of the core (#1916) 2024-08-09 19:07:32 -07:00
KeywordList.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Main.cs perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
MessageType.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Module.cs fix: Fixes sending packets and sidesteps a major issue with stackalloc and PGO in .NET 8 (#1607) 2023-11-21 12:18:20 -08:00
Party.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Poison.cs feat: Refactor Poison system, implement Darkglow & Parasitic effects (#2385) 2026-03-21 21:27:22 -07:00
Prompt.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Race.cs fix: Fixes TryParse for Race (#1244) 2022-11-12 00:31:25 -08:00
SecureTrade.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serial.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Server.csproj perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
Skills.cs chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
TileData.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
TileList.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00