fix: stop the idle-sleep backoff tripping on healthy hosts (#2572)

## Problem

The late-wake detector added in #2559 suspends idle sleeping on perfectly healthy hosts. The visible symptom is this Warning firing periodically on stable machines:

> This host returned a 2ms idle wait at least 8ms late 2 time(s) in the last second; idle sleeping suspended for 5000ms

Demoting it to Debug would hide the symptom but not the cost: every one of those lines means the shard dropped idle sleeping for 5s and burned a full core for no reason. The detector is what was mis-tuned.

## Cause 1 — lateness was a count, not a rate

An idle loop performs **~400–500 sleeps per second** (2ms each, bounded by the 8ms wheel tick). The trip condition was `late > 1` across two consecutive one-second samples — a **0.4% tail-outlier rate**. A co-tenant burst, a page fault, or another process changing the system timer resolution clears that bar on a healthy host.

A host that genuinely cannot schedule the process — throttled burstable vCPU — returns *most* of its waits late. Signal and noise were two orders of magnitude apart, and the check sat in the noise.

Now gated on the proportion, with the absolute count kept as a floor:

```csharp
if (late <= _lateWakeThreshold)             { _consecutiveBadSamples = 0; return; }  // floor
if (late * 100 < sleeps * _lateWakePercent) { _consecutiveBadSamples = 0; return; }  // rate
```

New `server.lateWakePercent` (default `10`). The floor is what keeps a window with only a handful of sleeps from tripping on a meaningless percentage; `server.lateWakeThreshold` keeps its existing meaning.

## Cause 2 — GC pauses were charged to the host

`dev-docs/debugging-event-loop.md` already documents that the GC collects preferentially **during idle sleeps** — that is the natural pause point it looks for. So the detector was systematically measuring the GC's chosen pause point and billing it to the host's scheduler. Not an occasional coincidence; a designed-in one.

```csharp
var collections = GC.CollectionCount(1);
NetState.WaitForCompletion(requested);
...
if (elapsed - requested >= Timer.TickRate && GC.CollectionCount(1) == collections)
```

Gen1 (which counts gen2 with it) rather than gen0 — gen0 pauses don't approach the 8ms `TickRate` bar anyway, and gating on them would discard useful samples. The second read short-circuits behind the overshoot test, so the common path costs **one** `GC.CollectionCount` per sleep: an internal counter read, single-digit nanoseconds, ~500/sec.

## Cause 3 — every backoff logged at Warning

Tiered to the escalation that already existed, since a single suspension is recoverable and not something an operator can act on:

| Backoff | Level |
|---|---|
| 1–2 | `Debug` |
| 3–5 | `Warning` (now includes the sleep count and "for the Nth time running") |
| ceiling | `Error`, unchanged |
| recovery | `Information` (new) |

Each backoff doubles the suspension, so every line is already a distinct escalation step — no further rate limiting needed.

## Drive-by

The `BackoffResetAfterCleanMs` reset only ran on the path to a *new* backoff, making it unreachable for a host that recovered for good — such a host never cleared its escalation or re-armed `_loggedBackoffCeiling`. It now runs on every health sample, which is also what makes the new recovery line reachable.

## Testing

Full solution builds clean, 0 warnings. No tests added: the state is private static in `Core` coupled to `_tickCount` with no injection point, and nothing covered it before — adding a seam purely to test it seemed worse than the gap. Happy to add one if reviewers disagree.
This commit is contained in:
Kamron Batman 2026-08-13 19:58:03 -07:00 committed by GitHub
parent 9b35b39d0d
commit 240118340e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 115 additions and 54 deletions

View file

@ -27,9 +27,16 @@ No build changes needed. Three signals exist, all actionable:
| Signal | Meaning | Action |
|---|---|---|
| Startup error: *host cannot honour short waits* | No high-resolution timer and `timeBeginPeriod` failed. Very old or unusual Windows. | Nothing is wrong with the server; it spins and uses a full core. Upgrade the OS or accept the core. |
| Warning: *host returned a Nms idle wait late* + sleeping suspended | The OS did not reschedule the process promptly after a 12ms wait. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
| Warning: *host returned a Nms idle wait late … for the Nth time running* | The OS did not reschedule the process promptly after a 12ms wait, through several escalating backoffs. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
| Error: *keeps returning idle waits late and sleeping has backed off N times* | The escalation hit its 120s ceiling. The host is not going to recover. | As above, but stop waiting for it to settle. Logged once per degradation, re-armed after a clean minute. |
| Admin gump → Performance → *Event Loop* | `Healthy` / `Sleep suspended (host)` / `Spinning (configured)` / `Spinning - host cannot honor short waits` | Same as above; the last verdict is the startup error's state, not a config choice. |
The first two backoffs of any episode log at **Debug**, not Warning: a single suspension is
recoverable and not something an operator can act on. Raise the log level if you are chasing a
marginal host and want to see them. Late wakes that coincide with a gen1-or-higher GC are not
counted at all — the GC deliberately collects during idle sleeps, so its pauses land there by
design and are not the host's fault.
If none of these fired and the shard still feels laggy, the cause is work, GC, or something a
boot-time signal cannot see. Continue.