fix: stop the idle-sleep backoff tripping on healthy hosts (#2572)

## Problem

The late-wake detector added in #2559 suspends idle sleeping on perfectly healthy hosts. The visible symptom is this Warning firing periodically on stable machines:

> This host returned a 2ms idle wait at least 8ms late 2 time(s) in the last second; idle sleeping suspended for 5000ms

Demoting it to Debug would hide the symptom but not the cost: every one of those lines means the shard dropped idle sleeping for 5s and burned a full core for no reason. The detector is what was mis-tuned.

## Cause 1 — lateness was a count, not a rate

An idle loop performs **~400–500 sleeps per second** (2ms each, bounded by the 8ms wheel tick). The trip condition was `late > 1` across two consecutive one-second samples — a **0.4% tail-outlier rate**. A co-tenant burst, a page fault, or another process changing the system timer resolution clears that bar on a healthy host.

A host that genuinely cannot schedule the process — throttled burstable vCPU — returns *most* of its waits late. Signal and noise were two orders of magnitude apart, and the check sat in the noise.

Now gated on the proportion, with the absolute count kept as a floor:

```csharp
if (late <= _lateWakeThreshold)             { _consecutiveBadSamples = 0; return; }  // floor
if (late * 100 < sleeps * _lateWakePercent) { _consecutiveBadSamples = 0; return; }  // rate
```

New `server.lateWakePercent` (default `10`). The floor is what keeps a window with only a handful of sleeps from tripping on a meaningless percentage; `server.lateWakeThreshold` keeps its existing meaning.

## Cause 2 — GC pauses were charged to the host

`dev-docs/debugging-event-loop.md` already documents that the GC collects preferentially **during idle sleeps** — that is the natural pause point it looks for. So the detector was systematically measuring the GC's chosen pause point and billing it to the host's scheduler. Not an occasional coincidence; a designed-in one.

```csharp
var collections = GC.CollectionCount(1);
NetState.WaitForCompletion(requested);
...
if (elapsed - requested >= Timer.TickRate && GC.CollectionCount(1) == collections)
```

Gen1 (which counts gen2 with it) rather than gen0 — gen0 pauses don't approach the 8ms `TickRate` bar anyway, and gating on them would discard useful samples. The second read short-circuits behind the overshoot test, so the common path costs **one** `GC.CollectionCount` per sleep: an internal counter read, single-digit nanoseconds, ~500/sec.

## Cause 3 — every backoff logged at Warning

Tiered to the escalation that already existed, since a single suspension is recoverable and not something an operator can act on:

| Backoff | Level |
|---|---|
| 1–2 | `Debug` |
| 3–5 | `Warning` (now includes the sleep count and "for the Nth time running") |
| ceiling | `Error`, unchanged |
| recovery | `Information` (new) |

Each backoff doubles the suspension, so every line is already a distinct escalation step — no further rate limiting needed.

## Drive-by

The `BackoffResetAfterCleanMs` reset only ran on the path to a *new* backoff, making it unreachable for a host that recovered for good — such a host never cleared its escalation or re-armed `_loggedBackoffCeiling`. It now runs on every health sample, which is also what makes the new recovery line reachable.

## Testing

Full solution builds clean, 0 warnings. No tests added: the state is private static in `Core` coupled to `_tickCount` with no injection point, and nothing covered it before — adding a seam purely to test it seemed worse than the gap. Happy to add one if reviewers disagree.
This commit is contained in:
Kamron Batman 2026-08-13 19:58:03 -07:00 • committed by GitHub
parent 9b35b39d0d
commit 240118340e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 115 additions and 54 deletions

View file

@ -105,7 +105,8 @@ See the README for the full supported list. Two things are worth calling out:
| Setting | Default | Why change it |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | `0` never sleeps: ~98% of one core, but zero skipped timer slots and zero lag. The choice for a large shard on dedicated CPU that would rather spend a core than risk a late wake. Above `2` the wheel starts losing slots. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before idle sleeping backs off. Raise on a jittery host; set very high to disable the backoff. |
| `server.lateWakeThreshold` | `1` | Floor for the backoff: idle waits the host may return a full tick late, per second, before the rate test below applies at all. Raise on a jittery host; set very high to disable the backoff. |
| `server.lateWakePercent` | `10` | Share of a second's idle waits that must come back late before idle sleeping backs off. An idle loop sleeps hundreds of times a second, so a bare count cannot tell a few tail outliers from a host that never schedules the process — a genuinely bad host misses *most* of its waits. `0` leaves `lateWakeThreshold` in sole charge. |
| `world.useMultithreadedSaves` | `true` | Set `false` on 2-core hosts so saves do not contend with the game loop. |
| `pathfinding.prebakeMaps` | varies | Leave off on memory-constrained hosts; it peaks above 1 GB while baking. |
| `network.sendBufferSize` | 256 KB | Lower it if you are memory-bound with many connections. |