The late-wake detector suspended idle sleeping on stable hosts, which both
spammed a Warning and cost the shard several seconds of full-core spin each
time for no reason. The log level was the visible symptom; the detector was
the bug.
Three causes, all fixed:
Lateness was a bare count, not a rate. An idle loop performs ~400-500 sleeps
a second (2ms each, bounded by the 8ms wheel tick), and the trip condition
was more than one late wake per second across two consecutive samples. That
is a 0.4% tail-outlier rate -- reachable by a co-tenant burst, a page fault,
or another process changing the system timer resolution. A host that
genuinely cannot schedule the process returns *most* of its waits late, two
orders of magnitude away. Gate on the proportion (server.lateWakePercent,
default 10) and keep server.lateWakeThreshold as a floor for windows with
few sleeps, where a percentage means nothing.
GC pauses were charged to the host. The GC collects preferentially during
idle sleeps -- that is the natural pause point it looks for, as
dev-docs/debugging-event-loop.md already documents -- so its pauses landed
in the measurement by design. Sample GC.CollectionCount(1) either side of
the wait and skip the sample if a collection intervened. The second read
short-circuits behind the overshoot test, so the common path pays for one
counter read per sleep.
Every backoff logged at Warning. Tier it to the escalation that already
existed: Debug for the first two (recoverable, not actionable), Warning once
the host has survived several doublings, and the existing Error at the
ceiling. Adds an Information line when a clean streak clears an escalation.
Also moves the BackoffResetAfterCleanMs reset to run on every health sample
rather than only on the path to a new backoff, where it was unreachable for
a host that recovered for good -- such a host never cleared its escalation
or re-armed the ceiling Error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6a02d4dece