ModernUO/Projects/Server
Kamron Batman 240118340e
fix: stop the idle-sleep backoff tripping on healthy hosts (#2572)
## Problem

The late-wake detector added in #2559 suspends idle sleeping on perfectly healthy hosts. The visible symptom is this Warning firing periodically on stable machines:

> This host returned a 2ms idle wait at least 8ms late 2 time(s) in the last second; idle sleeping suspended for 5000ms

Demoting it to Debug would hide the symptom but not the cost: every one of those lines means the shard dropped idle sleeping for 5s and burned a full core for no reason. The detector is what was mis-tuned.

## Cause 1 — lateness was a count, not a rate

An idle loop performs **~400–500 sleeps per second** (2ms each, bounded by the 8ms wheel tick). The trip condition was `late > 1` across two consecutive one-second samples — a **0.4% tail-outlier rate**. A co-tenant burst, a page fault, or another process changing the system timer resolution clears that bar on a healthy host.

A host that genuinely cannot schedule the process — throttled burstable vCPU — returns *most* of its waits late. Signal and noise were two orders of magnitude apart, and the check sat in the noise.

Now gated on the proportion, with the absolute count kept as a floor:

```csharp
if (late <= _lateWakeThreshold)             { _consecutiveBadSamples = 0; return; }  // floor
if (late * 100 < sleeps * _lateWakePercent) { _consecutiveBadSamples = 0; return; }  // rate
```

New `server.lateWakePercent` (default `10`). The floor is what keeps a window with only a handful of sleeps from tripping on a meaningless percentage; `server.lateWakeThreshold` keeps its existing meaning.

## Cause 2 — GC pauses were charged to the host

`dev-docs/debugging-event-loop.md` already documents that the GC collects preferentially **during idle sleeps** — that is the natural pause point it looks for. So the detector was systematically measuring the GC's chosen pause point and billing it to the host's scheduler. Not an occasional coincidence; a designed-in one.

```csharp
var collections = GC.CollectionCount(1);
NetState.WaitForCompletion(requested);
...
if (elapsed - requested >= Timer.TickRate && GC.CollectionCount(1) == collections)
```

Gen1 (which counts gen2 with it) rather than gen0 — gen0 pauses don't approach the 8ms `TickRate` bar anyway, and gating on them would discard useful samples. The second read short-circuits behind the overshoot test, so the common path costs **one** `GC.CollectionCount` per sleep: an internal counter read, single-digit nanoseconds, ~500/sec.

## Cause 3 — every backoff logged at Warning

Tiered to the escalation that already existed, since a single suspension is recoverable and not something an operator can act on:

| Backoff | Level |
|---|---|
| 1–2 | `Debug` |
| 3–5 | `Warning` (now includes the sleep count and "for the Nth time running") |
| ceiling | `Error`, unchanged |
| recovery | `Information` (new) |

Each backoff doubles the suspension, so every line is already a distinct escalation step — no further rate limiting needed.

## Drive-by

The `BackoffResetAfterCleanMs` reset only ran on the path to a *new* backoff, making it unreachable for a host that recovered for good — such a host never cleared its escalation or re-armed `_loggedBackoffCeiling`. It now runs on every health sample, which is also what makes the new recovery line reachable.

## Testing

Full solution builds clean, 0 warnings. No tests added: the state is private static in `Core` coupled to `_tickCount` with no injection point, and nothing covered it before — adding a seam purely to test it seemed worse than the gap. Happy to add one if reviewers disagree.
2026-08-13 19:58:03 -07:00
..
Buffers feat(buffers): add :L lowercase format spec to RawInterpolatedStringHandler (#2440) 2026-05-03 18:24:57 -07:00
Client feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Collections feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) 2026-07-25 11:59:37 -07:00
Comparers fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Compression feat(pathfinding): .swb format v8 compact index (#3b) (#2471) 2026-06-07 01:22:20 -07:00
Configuration fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
Console fix(console): stop headless servers from pegging a CPU core (#2535) 2026-07-16 18:52:43 -07:00
ContextMenus fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Diagnostics fix: harden idle-sleep scheduling against bad config and misattributed saves (#2567) 2026-08-09 22:05:18 -07:00
Events perf(login): run password hashing on a parked worker thread (#2566) 2026-08-09 00:13:34 -07:00
Exceptions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
GarbageCollection chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Geometry fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Items fix: stop stack merges and splits from laundering PlayerConstructed (#2576) 2026-08-13 19:18:06 -07:00
Json fix(json): Rectangle3DConverter loses a z-level on write (#2506) 2026-06-25 23:42:49 -07:00
Localization feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
Maps fix: Fixes pathfinding prebake and pathfinding multi-fallthrough. (#2478) 2026-06-08 11:59:24 -07:00
Menus feat: Pre-Publish 14 Crafting (supersedes #2181, #2381) (#2476) 2026-06-07 20:27:22 -07:00
Migrations refactor: decompose mobile/corpse hair, delete VirtualHairInfo, fix removal serial (#2462) (#2463) 2026-06-06 14:33:28 -07:00
Mobiles fix: stop stack merges and splits from laundering PlayerConstructed (#2576) 2026-08-13 19:18:06 -07:00
Network fix: harden idle-sleep scheduling against bad config and misattributed saves (#2567) 2026-08-09 22:05:18 -07:00
PropertyList fix(opl): refuse property list invalidation raised from inside GetProperties (#2555) 2026-07-28 21:29:03 -07:00
Random fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Regions fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serialization perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Targeting chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
Text feat: Add zero-alloc interpolation handler to ValueStringBuilder, replace all StringBuilder usage (#2387) 2026-03-22 14:23:44 -07:00
TileMatrix feat(pathfinding): JSONL recorder + public bake helpers (#2449) 2026-05-06 13:06:28 -07:00
Timer perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
TimeZones fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Utilities fix: Removes unnecessary dictionary removal guards (#2565) 2026-08-08 11:50:01 -07:00
World perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
AssemblyHandler.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Attributes.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
CityInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Commands.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Effects.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EntityFinalizationTracker.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
EventLoopTasks.cs perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
ExpansionInfo.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
FeatureFlags.cs fix: Streamlines insurance. Insurance only executes when enabled. (#2550) 2026-07-26 09:47:49 -07:00
Guild.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
HuePicker.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
IAccount.cs fix: Fixes looking up accounts that were renamed. (#2078) 2025-01-18 17:11:43 -08:00
IEntity.cs perf(saves): eliminate world-save freeze bottlenecks (~9.5x faster freeze) (#2525) 2026-07-16 22:53:28 -07:00
Interfaces.cs feat: Add CanSpawnMobile overload with props Z-range support. (#2293) 2025-12-27 17:01:15 -08:00
IVirtualCheckGump.cs feat: Moves gumps out of the core (#1916) 2024-08-09 19:07:32 -07:00
KeywordList.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Main.cs fix: stop the idle-sleep backoff tripping on healthy hosts (#2572) 2026-08-13 19:58:03 -07:00
MessageType.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Module.cs fix: Fixes sending packets and sidesteps a major issue with stackalloc and PGO in .NET 8 (#1607) 2023-11-21 12:18:20 -08:00
Party.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Poison.cs feat: Refactor Poison system, implement Darkglow & Parasitic effects (#2385) 2026-03-21 21:27:22 -07:00
Prompt.cs fix: Cleans up core code (#1187) 2022-10-10 21:47:08 -07:00
Race.cs fix: Fixes TryParse for Race (#1244) 2022-11-12 00:31:25 -08:00
SecureTrade.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Serial.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Server.csproj perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559) 2026-08-09 13:24:59 -07:00
Skills.cs chore: Use var everywhere (#2294) 2025-12-27 16:47:28 -08:00
TileData.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
TileList.cs fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00