warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: stop the idle-sleep backoff tripping on healthy hosts

Browse source 
The late-wake detector suspended idle sleeping on stable hosts, which both
spammed a Warning and cost the shard several seconds of full-core spin each
time for no reason. The log level was the visible symptom; the detector was
the bug.

Three causes, all fixed:

Lateness was a bare count, not a rate. An idle loop performs ~400-500 sleeps
a second (2ms each, bounded by the 8ms wheel tick), and the trip condition
was more than one late wake per second across two consecutive samples. That
is a 0.4% tail-outlier rate -- reachable by a co-tenant burst, a page fault,
or another process changing the system timer resolution. A host that
genuinely cannot schedule the process returns *most* of its waits late, two
orders of magnitude away. Gate on the proportion (server.lateWakePercent,
default 10) and keep server.lateWakeThreshold as a floor for windows with
few sleeps, where a percentage means nothing.

GC pauses were charged to the host. The GC collects preferentially during
idle sleeps -- that is the natural pause point it looks for, as
dev-docs/debugging-event-loop.md already documents -- so its pauses landed
in the measurement by design. Sample GC.CollectionCount(1) either side of
the wait and skip the sample if a collection intervened. The second read
short-circuits behind the overshoot test, so the common path pays for one
counter read per sleep.

Every backoff logged at Warning. Tier it to the escalation that already
existed: Debug for the first two (recoverable, not actionable), Warning once
the host has survived several doublings, and the existing Error at the
ceiling. Adds an Information line when a clean streak clears an escalation.

Also moves the BackoffResetAfterCleanMs reset to run on every health sample
rather than only on the path to a new backoff, where it was unreachable for
a host that recovered for good -- such a host never cleared its escalation
or re-armed the ceiling Error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kamron Batman 2026-08-11 21:48:35 -07:00
parent 1bc83339bb
commit 6a02d4dece
No known key found for this signature in database
GPG key ID: 7D81DF26D9A5D94A
3 changed files with 105 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

9
dev-docs/debugging-event-loop.md
View file

@ -27,9 +27,16 @@ No build changes needed. Three signals exist, all actionable:
| Signal | Meaning | Action |
|---|---|---|
| Startup error: *host cannot honour short waits* | No high-resolution timer and `timeBeginPeriod` failed. Very old or unusual Windows. | Nothing is wrong with the server; it spins and uses a full core. Upgrade the OS or accept the core. |
| Warning: *host returned a Nms idle wait late* + sleeping suspended | The OS did not reschedule the process promptly after a 1–2ms wait. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
| Warning: *host returned a Nms idle wait late … for the Nth time running* | The OS did not reschedule the process promptly after a 1–2ms wait, through several escalating backoffs. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
| Error: *keeps returning idle waits late and sleeping has backed off N times* | The escalation hit its 120s ceiling. The host is not going to recover. | As above, but stop waiting for it to settle. Logged once per degradation, re-armed after a clean minute. |
| Admin gump → Performance → *Event Loop* | `Healthy` / `Sleep suspended (host)` / `Spinning (configured)` / `Spinning - host cannot honor short waits` | Same as above; the last verdict is the startup error's state, not a config choice. |
The first two backoffs of any episode log at **Debug**, not Warning: a single suspension is
recoverable and not something an operator can act on. Raise the log level if you are chasing a
marginal host and want to see them. Late wakes that coincide with a gen1-or-higher GC are not
counted at all — the GC deliberately collects during idle sleeps, so its pauses land there by
design and are not the host's fault.
If none of these fired and the shard still feels laggy, the cause is work, GC, or something a
boot-time signal cannot see. Continue.