6aedbbe2ef
The loop span through its body regardless of whether there was anything to do -- ~10% of a desktop core for an empty shard, ~70% of a small VPS core, and a process that never idles is exactly what burstable vCPU plans throttle. The loop now blocks in NetState.WaitForCompletion whenever every queue it drains is empty, waking on the next timer tick or the moment work arrives. Receive completions, new connections, and cross-thread LoopContext.Post (via IORingGroup 1.0.10's sticky Wake) are all in the wait set, so sleeping adds no latency to any of them; only timer-driven logic sees wheel lag, bounded by server.eventLoopIdleWaitMs (default 2ms, 0 = never sleep). Measured on a real world of 190k items / 33k mobiles: 10.4% of a core to 0.8-1.0%, with peak tick lag unchanged. Spin mode independently gained 7x the iterations per core from the ring's AcceptEx rework. Sleeping also gives the GC natural pause points, which the old spin loop denied it -- memory no longer climbs until a save forces a collection. A sleep is bounded by the next wheel turn, so a correctly honoured sleep can never miss a deadline; the only way sleeping harms the wheel is the host returning the wait late. That overshoot is measured on every sleep, and an escalating backoff (server.lateWakeThreshold) suspends sleeping when it persists -- server work like saves or heavy commands cannot trip it by construction. Hosts without high-resolution waits are detected once at startup and spin instead. The admin gump shows the verdict instead of the now-meaningless CPS figure, which is removed. Time accounting for diagnosis is compiled out of normal builds: build with -p:EventLoopProfiling=true to enable EventLoopProfiler (per-phase wall time, sleep overshoot, GC pauses, stolen-time residual, ~15min ring buffer) and the [LoopStats command with CSV dump. See dev-docs/debugging-event-loop.md for the diagnosis funnel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
115 lines
7 KiB
Markdown
115 lines
7 KiB
Markdown
# Debugging Event Loop Performance
|
||
|
||
How to diagnose "the server feels slow" — written for both humans and AI assistants. Follow the
|
||
funnel in order; most incidents resolve before the last step. Do not start with dotnet-trace.
|
||
|
||
## The model
|
||
|
||
Every second of the main thread's wall time goes to exactly one of four places:
|
||
|
||
1. **Work** — the loop's phases: mobile deltas, item deltas, timer callbacks (`Timer.Slice`),
|
||
network processing (`NetState.Slice`), posted tasks (`LoopContext`).
|
||
2. **Sleep** — idle blocking in `NetState.WaitForCompletion`, bounded by the next timer tick and
|
||
`server.eventLoopIdleWaitMs`.
|
||
3. **GC pauses** — land inside whichever phase (or sleep) was running.
|
||
4. **Stolen** — the host ran something else: hypervisor scheduling, noisy neighbors, CPU credit
|
||
throttling.
|
||
|
||
A sleep is bounded by the time to the next wheel turn, so **a correctly honoured sleep can never
|
||
cost a deadline**. The only way sleeping harms the game is the wait *returning late* — that is
|
||
stolen time, and the server measures it directly on every sleep.
|
||
|
||
## Step 0 — Read what production already tells you
|
||
|
||
No build changes needed. Three signals exist, all actionable:
|
||
|
||
| Signal | Meaning | Action |
|
||
|---|---|---|
|
||
| Startup error: *host cannot honour short waits* | No high-resolution timer and `timeBeginPeriod` failed. Very old or unusual Windows. | Nothing is wrong with the server; it spins and uses a full core. Upgrade the OS or accept the core. |
|
||
| Warning: *host returned a Nms idle wait late* + sleeping suspended | The OS did not reschedule the process promptly after a 1–2ms wait. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
|
||
| Admin gump → Performance → *Event Loop* | `Healthy` / `Sleep suspended (host)` / `Spinning (configured)` | Same as above. |
|
||
|
||
If none of these fired and the shard still feels laggy, the cause is work, GC, or something a
|
||
boot-time signal cannot see. Continue.
|
||
|
||
## Step 1 — Flip the profiling build
|
||
|
||
```
|
||
dotnet build -p:EventLoopProfiling=true
|
||
```
|
||
|
||
This compiles in `EventLoopProfiler` (Server) and the `[LoopStats` command (UOContent). Without
|
||
the flag every hook call site is removed by the compiler (`[Conditional]`), so there is nothing to
|
||
"turn off" in normal builds and no cost to leave the hooks in the code. The profiling build's own
|
||
overhead is a handful of timestamp reads per iteration — small enough to run for days while
|
||
hunting an intermittent problem.
|
||
|
||
**Capture a baseline first.** Run `[LoopStats` while the shard feels *fine* and keep the CSV. The
|
||
profiler also keeps ~15 minutes of history in memory, so if the problem is episodic you can wait
|
||
for an episode and the good minutes on either side are already recorded. Numbers without a
|
||
baseline are how RunUO's profiler became useless — always compare bad minutes to good minutes on
|
||
the same box, build, and world.
|
||
|
||
## Step 2 — Read the decomposition
|
||
|
||
`[LoopStats` prints the last minute and writes the full history CSV (one row per second). Match
|
||
the shape against these signatures:
|
||
|
||
| Signature | Diagnosis | Next step |
|
||
|---|---|---|
|
||
| One phase consistently hot (e.g. `TimerSlice` 40%/s) | Deep processing in that subsystem | Step 3 — find the culprit in that phase |
|
||
| All phases near zero, `stolen` high, `lateWakes` > 0 | Host is stealing CPU | Host problem; see step 0 actions |
|
||
| `gcPauseMs` high, gen2 counts rising | GC pressure — something is allocating heavily | Step 3 on the allocating phase, or dotnet-counters for alloc rate |
|
||
| Iterations ≫ sleeps while shard is idle | The loop is not sleeping: a queue never drains or a wake storm | Check `IsIdle` inputs; a stuck signal in the ring is the historical example |
|
||
| Sleeps ≈ iterations, each sleep ~0ms | Spurious wake storm | Ring backend issue; count `wakesIssued` vs actual cross-thread posts |
|
||
| Everything normal, complaint persists | Not the event loop | Look at the network path, client, or DB/save timing |
|
||
|
||
**Wheel lag vs player lag:** `wheelLagMaxMs` is how late timer callbacks fired. Receives are
|
||
handled the moment they arrive (they wake the loop), so player-felt lag with a clean wheel points
|
||
away from the loop entirely.
|
||
|
||
## Step 3 — Find the culprit inside a hot phase
|
||
|
||
Add a temporary culprit hook rather than reaching for a tracer. The pattern: same
|
||
`[Conditional("EVENT_LOOP_PROFILING")]` attribute, own file or the profiler file, record only the
|
||
worst offender per second (identity + duration), never a per-event log. Examples:
|
||
|
||
- `TimerSlice` hot → time each timer callback, keep the max and its `timer.ToString()`.
|
||
- `NetworkSlice` hot → time packet handlers by packet id, keep the max.
|
||
- GC pressure → `dotnet-counters monitor --counters System.Runtime` for alloc rate first; it is
|
||
cheap and often names the culprit generation without a trace.
|
||
|
||
Keep the hook after the hunt if it earns its cost in the profiling build; delete it otherwise.
|
||
|
||
## Step 4 — dotnet-trace, last and targeted
|
||
|
||
Only when a hot phase resists the culprit hook. Know the costs: EventPipe visibly slows the
|
||
process (worst exactly when things are already bad) and adds artifacts to the trace — on small
|
||
vCPU hosts the tracer's own threads appear as hotspots and Rider/PerfView hotspot views can
|
||
mislead. Mitigate by being narrow:
|
||
|
||
- Trace the specific minutes the decomposition flagged, not "a while".
|
||
- `dotnet-trace collect --profile cpu-sampling --duration 00:00:30` is usually enough.
|
||
- Compare against a trace of a good minute (same rule as step 1: no baseline, no conclusions).
|
||
|
||
## The RAM / GC misconception (read before declaring a leak)
|
||
|
||
ModernUO allocates very little, and the GC collects opportunistically — mostly during idle sleeps
|
||
and world saves. Under a spinning loop (`eventLoopIdleWaitMs=0`, or the pre-2026 default) the GC
|
||
may find **no** natural pause point: memory climbs to a large fraction of physical RAM, a forced
|
||
collection eventually drops part of it, and fragmentation keeps the baseline permanently above
|
||
where it started. Task manager shows alarming numbers; the in-game numbers do not. **Performance
|
||
is unaffected — this is lazy collection working as designed, not a leak.** Idle sleeping largely
|
||
removes the effect because every sleep is a natural GC opportunity. Before investigating "a leak":
|
||
check `gen0/1/2` and `gcPauseMs` in the decomposition, and compare working set *after a world
|
||
save*, which forces the collection the spin loop never allowed.
|
||
|
||
## Rules of thumb
|
||
|
||
- Never trade always-on profiling for the numbers. Production carries one timestamp per sleep and
|
||
nothing else; everything heavier lives behind the build flag or on the `measure/event-loop`
|
||
branch (full harness, A/B scripts, vendored ring experiments).
|
||
- One decomposition chart beats a thousand log lines. Resist adding warnings the reader cannot
|
||
act on; the three production signals are deliberate.
|
||
- When filing or reporting: attach the baseline CSV and the episode CSV. Relative statements
|
||
("TimerSlice went from 4% to 61% during the episode") are the useful form.
|