fix: world save failure handling — no partial snapshots, no worker crash, staged publication (#2612)
Three pre-existing world-save failure-handling bugs, found while reviewing the delta-saves engine (#2610), split out so they land and get adopted on their own before the larger change. #2610 will be rebased on top once this merges. Tracked in project Delta Saves (#7). ## 1. A snapshot missing a segment was published `GenericEntityPersistence.WriteSnapshot` logged a segment (or self-payload) write error and carried on. The save was published without those records, and every entity in the failed segment was deleted at the next load. The error now propagates: `WriteFiles` never moves a partial snapshot over `Saves/`, the previous save stays authoritative, and staff are told. ## 2. A serializer exception killed the process, or published a partial freeze An entity whose `Serialize` threw on a worker thread was an unhandled exception on that thread, which terminates the process. On the inline (main-thread) drain it was caught, but the snapshot was still queued for writing with whatever the workers had produced. Workers now record the first exception and keep draining so the queue empties and the wake/pause handshake completes; after every worker paused, `Snapshot` treats a recorded error (or a failure of the drain itself) as a failed save: nothing is written, the previous save stays, staff are told, and the world resumes. A `WorldSave` handler throwing is logged but does not invalidate the serialized snapshot. ## 3. Publication was not transactional Publishing moved the previous `Saves/` away (`AutoArchive` in `WorldSavePostSnapshot`) and only then moved the new files in. A subscriber throwing after the archive, or the final move failing, left nothing at `Saves/` until the next save; a crash in that window lost the newest save at the next boot. The snapshot now moves to `Saves.next` as soon as its files are complete; only then do subscribers archive the previous save (same event, same `OldSavePath`, so `AutoArchive` and shard subscribers are unchanged), and the staged directory is renamed into place, which is atomic on one volume (file-by-file move across volumes). A staged directory is by construction a complete save newer than `Saves/`, so an interrupted publish is finished at the next boot (before load) or before the next save: whatever sits at `Saves/` is set aside as `Saves.previous-<timestamp>` (never deleted) and the staged save is published, with a warning naming the directory to archive or delete by hand. ## Tests Server.Tests 860 / UOContent.Tests green. New: a throwing serializer is recorded on the worker and the next drain starts clean; a segment that cannot be indexed fails `WriteSnapshot` instead of dropping records; staged-save recovery with and without an existing `Saves/`, and as a no-op.
This commit is contained in:
parent
84153fba58
commit
003491472f
5 changed files with 390 additions and 21 deletions
|
|
@ -78,6 +78,12 @@ public class SerializationThreadWorker
|
|||
internal List<int> Lengths => _lengths;
|
||||
internal List<IGenericSerializable> BufferEntities => _bufferEntities;
|
||||
|
||||
/// <summary>
|
||||
/// First serializer exception during the drain. The drain continues so the handshake
|
||||
/// completes; the loop fails the save once every worker has paused.
|
||||
/// </summary>
|
||||
public Exception Error { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Releases the write logs after the snapshot is written so serialized entity
|
||||
/// references don't linger between saves. Capacity is retained: the logs regrow to
|
||||
|
|
@ -154,6 +160,25 @@ public class SerializationThreadWorker
|
|||
public ReadOnlySpan<byte> GetHeap(int start, int length) => _heap.AsSpan(start, length);
|
||||
|
||||
private long ProcessChunk(in SerializationChunkSource.Chunk chunk, BufferWriter writer)
|
||||
{
|
||||
try
|
||||
{
|
||||
return ProcessChunkCore(in chunk, writer);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Error ??= ex;
|
||||
|
||||
if (chunk.Buffer != null)
|
||||
{
|
||||
_chunkSource.Return(chunk.Buffer, chunk.Count);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
private long ProcessChunkCore(in SerializationChunkSource.Chunk chunk, BufferWriter writer)
|
||||
{
|
||||
if (chunk.Single != null)
|
||||
{
|
||||
|
|
@ -213,6 +238,7 @@ public class SerializationThreadWorker
|
|||
public void DrainInline()
|
||||
{
|
||||
ReleaseWriteLogs();
|
||||
Error = null;
|
||||
|
||||
var writer = new BufferWriter(_heap, true);
|
||||
var entities = 0L;
|
||||
|
|
@ -238,6 +264,7 @@ public class SerializationThreadWorker
|
|||
while (worker._startEvent.WaitOne())
|
||||
{
|
||||
worker.ReleaseWriteLogs();
|
||||
worker.Error = null;
|
||||
|
||||
var writer = new BufferWriter(worker._heap, true);
|
||||
var entities = 0L;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue