feat: Implements new robust/pluggable backup/archive system. (#2388)

## Summary

Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.

### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics

### What changed

**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.

**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).

**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.

**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).

**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.

**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.

### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule

### Files

| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |

### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup

### Plugin example (external repo)
```csharp
public static class S3Plugin
{
    public static void Configure()
    {
        ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
    }
}
```

## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach
This commit is contained in:
Kamron Batman 2026-03-22 19:51:20 -07:00 committed by GitHub
parent 61e41df00c
commit 3b4e1137f6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 2421 additions and 615 deletions

View file

@ -0,0 +1,279 @@
/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: ManagedArchive.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Collections.Generic;
using System.Formats.Tar;
using System.IO;
using System.IO.Compression;
using Server.Logging;
using ZstdNet;
namespace Server.Compression;
/// <summary>
/// Cross-platform archive creation and extraction using System.Formats.Tar + ZstdNet (native libzstd).
/// Single-pass streaming: TarWriter → CompressionStream → FileStream.
/// </summary>
public static class ManagedArchive
{
private static readonly ILogger logger = LogFactory.GetLogger(typeof(ManagedArchive));
/// <summary>
/// Creates a .tar.zst archive from one or more source directories.
/// Writes to a temp file first, then atomically renames to the destination.
/// </summary>
/// <param name="sourcePaths">Directories to include in the archive.</param>
/// <param name="destinationFile">Final .tar.zst file path.</param>
/// <param name="relativeTo">Base path for computing relative entry names.</param>
/// <param name="compressionLevel">ZSTD compression level (1-22, default 3).</param>
/// <returns>The number of entries written, or -1 on failure.</returns>
public static int CreateTarZstd(
IEnumerable<string> sourcePaths,
string destinationFile,
string relativeTo,
int compressionLevel = 3)
{
var tempFile = destinationFile + ".tmp";
try
{
new FileInfo(destinationFile).EnsureDirectory();
using var fileStream = new FileStream(
tempFile,
FileMode.Create,
FileAccess.Write,
FileShare.None,
bufferSize: 81920
);
using var compressionOptions = new CompressionOptions(compressionLevel);
using var zstdStream = new CompressionStream(fileStream, compressionOptions);
using var tarWriter = new TarWriter(zstdStream, TarEntryFormat.Pax, leaveOpen: true);
var entryCount = 0;
long totalBytes = 0;
foreach (var sourcePath in sourcePaths)
{
if (!Directory.Exists(sourcePath))
{
logger.Warning("Source path does not exist, skipping: {Path}", sourcePath);
continue;
}
var sourceDir = new DirectoryInfo(sourcePath);
var relativeName = Path.GetRelativePath(relativeTo, sourcePath);
foreach (var file in sourceDir.EnumerateFiles("*", SearchOption.AllDirectories))
{
var entryName = Path.Combine(relativeName, Path.GetRelativePath(sourcePath, file.FullName))
.Replace('\\', '/');
tarWriter.WriteEntry(file.FullName, entryName);
entryCount++;
totalBytes += file.Length;
if (entryCount % 100 == 0)
{
logger.Debug(
"Archive progress: {Count} entries, {Size:F1} MB",
entryCount,
totalBytes / 1048576.0
);
}
}
}
// Flush tar → zstd → file
tarWriter.Dispose();
zstdStream.Dispose();
fileStream.Dispose();
// Atomic rename on same filesystem
if (File.Exists(destinationFile))
{
File.Delete(destinationFile);
}
File.Move(tempFile, destinationFile);
logger.Information(
"Created archive at {Path} ({Count} entries, {Size:F1} MB)",
destinationFile,
entryCount,
totalBytes / 1048576.0
);
return entryCount;
}
catch (Exception ex)
{
logger.Error(ex, "Failed to create archive at {Path}", destinationFile);
try
{
if (File.Exists(tempFile))
{
File.Delete(tempFile);
}
}
catch
{
// Best-effort cleanup
}
return -1;
}
}
/// <summary>
/// Extracts a .tar.zst archive to a directory.
/// </summary>
public static bool ExtractTarZstd(string archiveFile, string outputDirectory)
{
try
{
Directory.CreateDirectory(outputDirectory);
using var fileStream = new FileStream(
archiveFile,
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 81920
);
using var zstdStream = new DecompressionStream(fileStream);
return ExtractTarFromStream(zstdStream, outputDirectory);
}
catch (Exception ex)
{
logger.Error(ex, "Failed to extract archive {File}", archiveFile);
return false;
}
}
/// <summary>
/// Extracts a .tar.gz archive to a directory.
/// </summary>
public static bool ExtractTarGz(string archiveFile, string outputDirectory)
{
try
{
Directory.CreateDirectory(outputDirectory);
using var fileStream = new FileStream(
archiveFile,
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 81920
);
using var gzStream = new GZipStream(fileStream, CompressionMode.Decompress);
return ExtractTarFromStream(gzStream, outputDirectory);
}
catch (Exception ex)
{
logger.Error(ex, "Failed to extract archive {File}", archiveFile);
return false;
}
}
/// <summary>
/// Extracts a plain .tar archive to a directory.
/// </summary>
public static bool ExtractTar(string archiveFile, string outputDirectory)
{
try
{
Directory.CreateDirectory(outputDirectory);
using var fileStream = new FileStream(
archiveFile,
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 81920
);
return ExtractTarFromStream(fileStream, outputDirectory);
}
catch (Exception ex)
{
logger.Error(ex, "Failed to extract archive {File}", archiveFile);
return false;
}
}
private static bool ExtractTarFromStream(Stream stream, string outputDirectory)
{
using var tarReader = new TarReader(stream);
while (tarReader.GetNextEntry() is { } entry)
{
var destPath = Path.Combine(outputDirectory, entry.Name);
var destDir = Path.GetDirectoryName(destPath);
if (destDir != null)
{
Directory.CreateDirectory(destDir);
}
if (entry.EntryType is TarEntryType.RegularFile or TarEntryType.V7RegularFile)
{
entry.ExtractToFile(destPath, overwrite: true);
}
}
return true;
}
/// <summary>
/// Counts entries in a .tar.zst archive without extracting content.
/// Used for verification after archive creation.
/// </summary>
public static int CountEntries(string archiveFile)
{
try
{
using var fileStream = new FileStream(
archiveFile,
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 81920
);
using var zstdStream = new DecompressionStream(fileStream);
using var tarReader = new TarReader(zstdStream);
var count = 0;
while (tarReader.GetNextEntry() is not null)
{
count++;
}
return count;
}
catch (Exception ex)
{
logger.Error(ex, "Failed to count entries in archive {File}", archiveFile);
return -1;
}
}
}