ModernUO/dev-docs
Kamron Batman 240118340e
fix: stop the idle-sleep backoff tripping on healthy hosts (#2572)
## Problem

The late-wake detector added in #2559 suspends idle sleeping on perfectly healthy hosts. The visible symptom is this Warning firing periodically on stable machines:

> This host returned a 2ms idle wait at least 8ms late 2 time(s) in the last second; idle sleeping suspended for 5000ms

Demoting it to Debug would hide the symptom but not the cost: every one of those lines means the shard dropped idle sleeping for 5s and burned a full core for no reason. The detector is what was mis-tuned.

## Cause 1 — lateness was a count, not a rate

An idle loop performs **~400–500 sleeps per second** (2ms each, bounded by the 8ms wheel tick). The trip condition was `late > 1` across two consecutive one-second samples — a **0.4% tail-outlier rate**. A co-tenant burst, a page fault, or another process changing the system timer resolution clears that bar on a healthy host.

A host that genuinely cannot schedule the process — throttled burstable vCPU — returns *most* of its waits late. Signal and noise were two orders of magnitude apart, and the check sat in the noise.

Now gated on the proportion, with the absolute count kept as a floor:

```csharp
if (late <= _lateWakeThreshold)             { _consecutiveBadSamples = 0; return; }  // floor
if (late * 100 < sleeps * _lateWakePercent) { _consecutiveBadSamples = 0; return; }  // rate
```

New `server.lateWakePercent` (default `10`). The floor is what keeps a window with only a handful of sleeps from tripping on a meaningless percentage; `server.lateWakeThreshold` keeps its existing meaning.

## Cause 2 — GC pauses were charged to the host

`dev-docs/debugging-event-loop.md` already documents that the GC collects preferentially **during idle sleeps** — that is the natural pause point it looks for. So the detector was systematically measuring the GC's chosen pause point and billing it to the host's scheduler. Not an occasional coincidence; a designed-in one.

```csharp
var collections = GC.CollectionCount(1);
NetState.WaitForCompletion(requested);
...
if (elapsed - requested >= Timer.TickRate && GC.CollectionCount(1) == collections)
```

Gen1 (which counts gen2 with it) rather than gen0 — gen0 pauses don't approach the 8ms `TickRate` bar anyway, and gating on them would discard useful samples. The second read short-circuits behind the overshoot test, so the common path costs **one** `GC.CollectionCount` per sleep: an internal counter read, single-digit nanoseconds, ~500/sec.

## Cause 3 — every backoff logged at Warning

Tiered to the escalation that already existed, since a single suspension is recoverable and not something an operator can act on:

| Backoff | Level |
|---|---|
| 1–2 | `Debug` |
| 3–5 | `Warning` (now includes the sleep count and "for the Nth time running") |
| ceiling | `Error`, unchanged |
| recovery | `Information` (new) |

Each backoff doubles the suspension, so every line is already a distinct escalation step — no further rate limiting needed.

## Drive-by

The `BackoffResetAfterCleanMs` reset only ran on the path to a *new* backoff, making it unreachable for a host that recovered for good — such a host never cleared its escalation or re-armed `_loggedBackoffCeiling`. It now runs on every health sample, which is also what makes the new recovery line reachable.

## Testing

Full solution builds clean, 0 warnings. No tests added: the state is private static in `Core` coupled to `_tickCount` with no injection point, and nothing covered it before — adding a seam purely to test it seemed worse than the gap. Happy to add one if reviewers disagree.
2026-08-13 19:58:03 -07:00
..
claude-skills perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
runuo-migration-docs fix: Harden Advanced Search: crash-safety, autosave, correct results & worker fixes (#2543) 2026-07-21 07:51:06 -07:00
code-standards.md feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
commands-targeting.md chore: Adds AI instructions and SKILLs for ModernUO codebase (#2347) 2026-03-01 11:42:19 -08:00
configuration.md fix: Fixes Argon2 verify correctness and the password upgrade lockout (#2562) 2026-08-08 00:24:59 -07:00
content-patterns.md docs: Updates CLAUDE dev-docs/skills for serialization (#2372) 2026-03-15 01:05:03 -07:00
debugging-event-loop.md fix: stop the idle-sleep backoff tripping on healthy hosts (#2572) 2026-08-13 19:58:03 -07:00
era-expansion.md chore: Adds AI instructions and SKILLs for ModernUO codebase (#2347) 2026-03-01 11:42:19 -08:00
event-scheduler.md chore: Adds AI instructions and SKILLs for ModernUO codebase (#2347) 2026-03-01 11:42:19 -08:00
events.md chore: Adds AI instructions and SKILLs for ModernUO codebase (#2347) 2026-03-01 11:42:19 -08:00
gump-system.md docs(messages): document interpolation anti-patterns and :L format spec (#2441) 2026-05-03 18:23:50 -07:00
ip-bans-and-allowlists.md feat(network): allowlist false-positive IPs, escalate on behavior (#2556) 2026-07-30 23:12:17 -07:00
networking-packets.md feat(network): allowlist false-positive IPs, escalate on behavior (#2556) 2026-07-30 23:12:17 -07:00
pathfinding.md fix(pathfinding): stop opening every .swb twice at boot (#2548) 2026-07-25 13:07:20 -07:00
platform-prerequisites.md fix: Require only runtime packages on Linux, and check ICU and tzdata the way the runtime does (#2561) 2026-08-07 15:03:08 -07:00
property-lists.md fix(opl): refuse property list invalidation raised from inside GetProperties (#2555) 2026-07-28 21:29:03 -07:00
regions.md chore: Adds AI instructions and SKILLs for ModernUO codebase (#2347) 2026-03-01 11:42:19 -08:00
serialization.md fix: Preserve corpse notoriety across server restart (#2426) 2026-05-03 02:15:01 -07:00
server-lifecycle.md refactor(server): unify first-boot prompts into the ConfigurePrompts phase (#2477) 2026-06-08 05:18:46 -07:00
server-requirements.md fix: stop the idle-sleep backoff tripping on healthy hosts (#2572) 2026-08-13 19:58:03 -07:00
string-handling.md docs(messages): document interpolation anti-patterns and :L format spec (#2441) 2026-05-03 18:23:50 -07:00
t2a-crafting.md feat: Pre-Publish 14 Crafting (supersedes #2181, #2381) (#2476) 2026-06-07 20:27:22 -07:00
threading-model.md perf(login): run password hashing on a parked worker thread (#2566) 2026-08-09 00:13:34 -07:00
tick-counts.md perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
timers.md docs: Updates CLAUDE dev-docs/skills for serialization (#2372) 2026-03-15 01:05:03 -07:00