fix(saves): close a deadlock race in the worker pause handshake

Execute() signaled _stopEvent before clearing _pause and before checking the
exit flag. Wake/Sleep/Exit all run on the owning thread, so the moment
_stopEvent is set the owner can start another pause cycle - Exit() does
exactly that. Two interleavings in that window are fatal:

- The worker's late "_pause = false" clobbers the new cycle's pause request,
  so the worker consumes the wake and spins forever waiting for a pause that
  never reads true while the owner blocks in Sleep().
- The worker's late exit check observes the _exit flag written by an Exit()
  whose Sleep() is still pending, and returns without ever signaling it.

Either way the run hangs silently. The ordering predates this branch, but
production callers never sat inside the window (one Wake/Sleep pair per
save, minutes apart; Exit only long after the last handshake). The tests
added on this branch are the first callers that run pause cycles
back-to-back, and a saturated CI runner hit the window: this is what
stalled the CentOS job for 2h41m. The race is not rare under that pattern -
a churn test reproduces the deadlock in 4 out of 4 runs locally.

Fix: sample the exit condition and clear _pause (volatile) BEFORE signaling
_stopEvent. A post-signal Exit() is then always serviced by one more
wake/drain/signal round, and its pause request can no longer be clobbered.
Drain-before-exit semantics on Core.Closing are unchanged.

Adds a watchdog churn regression test (2000 wake/sleep/exit lifecycles) that
fails instead of hanging if the race is reintroduced: red 4/4 deadlocks
without the fix, green 4/4 at ~550ms with it. Server.Tests 789/789 and
UOContent.Tests 509/509 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Kamron Batman 2026-07-16 22:35:06 -07:00
parent 5b74a346df
commit 95b225aef7
No known key found for this signature in database
GPG key ID: 7D81DF26D9A5D94A
2 changed files with 70 additions and 3 deletions

View file

@ -270,10 +270,18 @@ public class SerializationThreadWorker
writer.Close();
worker._stopEvent.Set(); // Allow the main thread to continue now that we are finished
worker._pause = false;
// Wake/Sleep/Exit all run on the owning thread, so the moment _stopEvent is set
// the owner may start another pause cycle (Exit does exactly that). Clear _pause
// and sample the exit condition BEFORE signaling: clearing after the signal can
// clobber the next cycle's pause request (this thread then spins forever waiting
// for a pause that never reads true), and deciding to exit after the signal can
// return without servicing that cycle's Sleep (the owner then blocks forever).
var exiting = Core.Closing || worker._exit;
Volatile.Write(ref worker._pause, false);
if (Core.Closing || worker._exit)
worker._stopEvent.Set(); // Allow the main thread to continue now that we are finished
if (exiting)
{
return;
}