feat: Implements new robust/pluggable backup/archive system. (#2388)

## Summary

Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.

### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics

### What changed

**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.

**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).

**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.

**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).

**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.

**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.

### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule

### Files

| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |

### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup

### Plugin example (external repo)
```csharp
public static class S3Plugin
{
    public static void Configure()
    {
        ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
    }
}
```

## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach

Projects/UOContent/Compression/ManagedArchive.cs

@@ -0,0 +1,279 @@
+/*************************************************************************
+ * ModernUO                                                              *
+ * Copyright 2019-2026 - ModernUO Development Team                       *
+ * Email: hi@modernuo.com                                                *
+ * File: ManagedArchive.cs                                               *
+ *                                                                       *
+ * This program is free software: you can redistribute it and/or modify  *
+ * it under the terms of the GNU General Public License as published by  *
+ * the Free Software Foundation, either version 3 of the License, or     *
+ * (at your option) any later version.                                   *
+ *                                                                       *
+ * You should have received a copy of the GNU General Public License     *
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>. *
+ *************************************************************************/
+
+using System;
+using System.Collections.Generic;
+using System.Formats.Tar;
+using System.IO;
+using System.IO.Compression;
+using Server.Logging;
+using ZstdNet;
+
+namespace Server.Compression;
+
+/// <summary>
+/// Cross-platform archive creation and extraction using System.Formats.Tar + ZstdNet (native libzstd).
+/// Single-pass streaming: TarWriter → CompressionStream → FileStream.
+/// </summary>
+public static class ManagedArchive
+{
+    private static readonly ILogger logger = LogFactory.GetLogger(typeof(ManagedArchive));
+
+    /// <summary>
+    /// Creates a .tar.zst archive from one or more source directories.
+    /// Writes to a temp file first, then atomically renames to the destination.
+    /// </summary>
+    /// <param name="sourcePaths">Directories to include in the archive.</param>
+    /// <param name="destinationFile">Final .tar.zst file path.</param>
+    /// <param name="relativeTo">Base path for computing relative entry names.</param>
+    /// <param name="compressionLevel">ZSTD compression level (1-22, default 3).</param>
+    /// <returns>The number of entries written, or -1 on failure.</returns>
+    public static int CreateTarZstd(
+        IEnumerable<string> sourcePaths,
+        string destinationFile,
+        string relativeTo,
+        int compressionLevel = 3)
+    {
+        var tempFile = destinationFile + ".tmp";
+
+        try
+        {
+            new FileInfo(destinationFile).EnsureDirectory();
+
+            using var fileStream = new FileStream(
+                tempFile,
+                FileMode.Create,
+                FileAccess.Write,
+                FileShare.None,
+                bufferSize: 81920
+            );
+
+            using var compressionOptions = new CompressionOptions(compressionLevel);
+            using var zstdStream = new CompressionStream(fileStream, compressionOptions);
+            using var tarWriter = new TarWriter(zstdStream, TarEntryFormat.Pax, leaveOpen: true);
+
+            var entryCount = 0;
+            long totalBytes = 0;
+
+            foreach (var sourcePath in sourcePaths)
+            {
+                if (!Directory.Exists(sourcePath))
+                {
+                    logger.Warning("Source path does not exist, skipping: {Path}", sourcePath);
+                    continue;
+                }
+
+                var sourceDir = new DirectoryInfo(sourcePath);
+                var relativeName = Path.GetRelativePath(relativeTo, sourcePath);
+
+                foreach (var file in sourceDir.EnumerateFiles("*", SearchOption.AllDirectories))
+                {
+                    var entryName = Path.Combine(relativeName, Path.GetRelativePath(sourcePath, file.FullName))
+                        .Replace('\\', '/');
+
+                    tarWriter.WriteEntry(file.FullName, entryName);
+                    entryCount++;
+                    totalBytes += file.Length;
+
+                    if (entryCount % 100 == 0)
+                    {
+                        logger.Debug(
+                            "Archive progress: {Count} entries, {Size:F1} MB",
+                            entryCount,
+                            totalBytes / 1048576.0
+                        );
+                    }
+                }
+            }
+
+            // Flush tar → zstd → file
+            tarWriter.Dispose();
+            zstdStream.Dispose();
+            fileStream.Dispose();
+
+            // Atomic rename on same filesystem
+            if (File.Exists(destinationFile))
+            {
+                File.Delete(destinationFile);
+            }
+
+            File.Move(tempFile, destinationFile);
+
+            logger.Information(
+                "Created archive at {Path} ({Count} entries, {Size:F1} MB)",
+                destinationFile,
+                entryCount,
+                totalBytes / 1048576.0
+            );
+
+            return entryCount;
+        }
+        catch (Exception ex)
+        {
+            logger.Error(ex, "Failed to create archive at {Path}", destinationFile);
+
+            try
+            {
+                if (File.Exists(tempFile))
+                {
+                    File.Delete(tempFile);
+                }
+            }
+            catch
+            {
+                // Best-effort cleanup
+            }
+
+            return -1;
+        }
+    }
+
+    /// <summary>
+    /// Extracts a .tar.zst archive to a directory.
+    /// </summary>
+    public static bool ExtractTarZstd(string archiveFile, string outputDirectory)
+    {
+        try
+        {
+            Directory.CreateDirectory(outputDirectory);
+
+            using var fileStream = new FileStream(
+                archiveFile,
+                FileMode.Open,
+                FileAccess.Read,
+                FileShare.Read,
+                bufferSize: 81920
+            );
+
+            using var zstdStream = new DecompressionStream(fileStream);
+            return ExtractTarFromStream(zstdStream, outputDirectory);
+        }
+        catch (Exception ex)
+        {
+            logger.Error(ex, "Failed to extract archive {File}", archiveFile);
+            return false;
+        }
+    }
+
+    /// <summary>
+    /// Extracts a .tar.gz archive to a directory.
+    /// </summary>
+    public static bool ExtractTarGz(string archiveFile, string outputDirectory)
+    {
+        try
+        {
+            Directory.CreateDirectory(outputDirectory);
+
+            using var fileStream = new FileStream(
+                archiveFile,
+                FileMode.Open,
+                FileAccess.Read,
+                FileShare.Read,
+                bufferSize: 81920
+            );
+
+            using var gzStream = new GZipStream(fileStream, CompressionMode.Decompress);
+            return ExtractTarFromStream(gzStream, outputDirectory);
+        }
+        catch (Exception ex)
+        {
+            logger.Error(ex, "Failed to extract archive {File}", archiveFile);
+            return false;
+        }
+    }
+
+    /// <summary>
+    /// Extracts a plain .tar archive to a directory.
+    /// </summary>
+    public static bool ExtractTar(string archiveFile, string outputDirectory)
+    {
+        try
+        {
+            Directory.CreateDirectory(outputDirectory);
+
+            using var fileStream = new FileStream(
+                archiveFile,
+                FileMode.Open,
+                FileAccess.Read,
+                FileShare.Read,
+                bufferSize: 81920
+            );
+
+            return ExtractTarFromStream(fileStream, outputDirectory);
+        }
+        catch (Exception ex)
+        {
+            logger.Error(ex, "Failed to extract archive {File}", archiveFile);
+            return false;
+        }
+    }
+
+    private static bool ExtractTarFromStream(Stream stream, string outputDirectory)
+    {
+        using var tarReader = new TarReader(stream);
+
+        while (tarReader.GetNextEntry() is { } entry)
+        {
+            var destPath = Path.Combine(outputDirectory, entry.Name);
+
+            var destDir = Path.GetDirectoryName(destPath);
+            if (destDir != null)
+            {
+                Directory.CreateDirectory(destDir);
+            }
+
+            if (entry.EntryType is TarEntryType.RegularFile or TarEntryType.V7RegularFile)
+            {
+                entry.ExtractToFile(destPath, overwrite: true);
+            }
+        }
+
+        return true;
+    }
+
+    /// <summary>
+    /// Counts entries in a .tar.zst archive without extracting content.
+    /// Used for verification after archive creation.
+    /// </summary>
+    public static int CountEntries(string archiveFile)
+    {
+        try
+        {
+            using var fileStream = new FileStream(
+                archiveFile,
+                FileMode.Open,
+                FileAccess.Read,
+                FileShare.Read,
+                bufferSize: 81920
+            );
+
+            using var zstdStream = new DecompressionStream(fileStream);
+            using var tarReader = new TarReader(zstdStream);
+
+            var count = 0;
+            while (tarReader.GetNextEntry() is not null)
+            {
+                count++;
+            }
+
+            return count;
+        }
+        catch (Exception ex)
+        {
+            logger.Error(ex, "Failed to count entries in archive {File}", archiveFile);
+            return -1;
+        }
+    }
+}

Projects/UOContent/Compression/TarArchive.cs

@@ -1,122 +0,0 @@
-using System;
-using System.Collections.Generic;
-using System.Diagnostics;
-using System.IO;
-using System.IO.Compression;
-using System.Net.Http;
-using Server.Text;
-
-namespace Server.Compression
-{
-    public static class TarArchive
-    {
-        private const string _libArchiveWindowsUrl = @"https://www.libarchive.org/downloads/libarchive-v3.5.2-win64.zip";
-        private static string _pathToTar;
-
-        private static string GetPathToTar()
-        {
-            if (!Core.IsWindows || File.Exists(@"C:\Windows\system32\tar.exe"))
-            {
-                return "tar";
-            }
-
-            var tarPath = Path.Combine(Core.BaseDirectory, "bsdtar/bsdtar.exe");
-            return File.Exists(tarPath) ? tarPath : DownloadTarForWindows();
-        }
-
-        private static int RunTar(string arguments, string compressionProgramPath)
-        {
-            var process = new Process
-            {
-                StartInfo = new ProcessStartInfo
-                {
-                    FileName = _pathToTar,
-                    Arguments = arguments
-                }
-            };
-
-            if (compressionProgramPath != null)
-            {
-                var envPaths = $"{process.StartInfo.EnvironmentVariables["PATH"]}:{compressionProgramPath}";
-                process.StartInfo.EnvironmentVariables["PATH"] = envPaths;
-            }
-
-            process.Start();
-            process.WaitForExit();
-
-            return process.ExitCode;
-        }
-
-        private static string DownloadTarForWindows()
-        {
-            var tempDir = PathUtility.EnsureRandomPath(Path.GetTempPath());
-
-            var libarchiveFile = Path.Combine(tempDir, "libarchive.zip");
-            // This isn't called often so we don't need to optimize
-            using (var hc = new HttpClient())
-            {
-                var result = hc.Send(new HttpRequestMessage(HttpMethod.Get, new Uri(_libArchiveWindowsUrl)));
-                using var stream = result.Content.ReadAsStream();
-                using var fs = new FileStream(libarchiveFile, FileMode.Create, FileAccess.Write, FileShare.None);
-                stream.CopyTo(fs);
-            }
-
-            ZipFile.ExtractToDirectory(libarchiveFile, tempDir);
-            var libArchivePath = Path.Combine(tempDir, "libarchive");
-            PathUtility.MoveDirectoryContents(Path.Combine(libArchivePath, "bin"), Path.Combine(Core.BaseDirectory, "bsdtar"));
-            File.Delete(libarchiveFile);
-
-            return Path.Combine(Core.BaseDirectory, "bsdtar/bsdtar.exe");
-        }
-
-        public static bool ExtractToDirectory(
-            string fileNamePath,
-            string outputDirectory,
-            string compressCommand = null,
-            string compressionProgramPath = null
-        )
-        {
-            _pathToTar ??= GetPathToTar();
-
-            var useExternalCompression = compressCommand != null ? $"--use-compress-program \"{compressCommand}\" " : "";
-            var arguments = $"{useExternalCompression} -xf \"{fileNamePath}\" -C \"{outputDirectory}\"";
-
-            return RunTar(arguments, compressionProgramPath) == 0;
-        }
-
-        public static bool CreateFromPaths(
-            IEnumerable<string> paths,
-            string destinationArchiveFileName,
-            string relativeTo,
-            string compressCommand = null,
-            string compressionProgramPath = null
-        )
-        {
-            _pathToTar ??= GetPathToTar();
-
-            new FileInfo(destinationArchiveFileName).EnsureDirectory();
-
-            using var sb = ValueStringBuilder.Create();
-            var i = 0;
-            foreach (var path in paths)
-            {
-                var relativePath = Path.GetRelativePath(relativeTo, path);
-                if (i++ > 0)
-                {
-                    sb.Append($" \"{relativePath}\"");
-                }
-                else
-                {
-                    sb.Append($"\"{relativePath}\"");
-                }
-            }
-            var pathsToCompress = sb.ToString();
-
-            var tarFlags = compressCommand == null ? "-acf" : "-cf";
-            var useExternalCompression = compressCommand != null ? $"--use-compress-program \"{compressCommand}\" " : "";
-            var arguments = $"{useExternalCompression}{tarFlags} \"{destinationArchiveFileName}\" -C \"{relativeTo}\" {pathsToCompress}";
-
-            return RunTar(arguments, compressionProgramPath) == 0;
-        }
-    }
-}

Projects/UOContent/Compression/ZstdArchive.cs

@@ -1,95 +0,0 @@
-using System.Collections.Generic;
-using System.Diagnostics;
-using System.IO;
-using System.Reflection;
-
-namespace Server.Compression
-{
-    public static class ZstdArchive
-    {
-        private static readonly string _pathToZstd = new FileInfo(Assembly.GetExecutingAssembly().Location).Directory?.FullName;
-
-        public static bool ExtractToDirectory(string fileNamePath, string outputDirectory)
-        {
-            // bsdtar has a bug and hangs, so we are doing it in two steps.
-            if (Core.IsWindows)
-            {
-                var tempDir = PathUtility.EnsureRandomPath(Path.GetTempPath());
-                var tempTarArchive = Path.Combine(tempDir, "temp-file.tar");
-
-                try
-                {
-                    var process = new Process
-                    {
-                        StartInfo = new ProcessStartInfo
-                        {
-                            FileName = Path.Combine(_pathToZstd, "zstd.exe"),
-                            Arguments = $"-q -d \"{fileNamePath}\" -o \"{tempTarArchive}\""
-                        }
-                    };
-
-                    process.Start();
-                    process.WaitForExit();
-
-                    return process.ExitCode == 0 && TarArchive.ExtractToDirectory(tempTarArchive, outputDirectory);
-                }
-                catch
-                {
-                    return false;
-                }
-                finally
-                {
-                    File.Delete(tempTarArchive);
-                }
-            }
-
-            return TarArchive.ExtractToDirectory(fileNamePath, outputDirectory, "zstd -q -d", _pathToZstd);
-        }
-
-        public static bool CreateFromPaths(
-            IEnumerable<string> paths,
-            string destinationArchiveFileName,
-            string relativeTo
-        )
-        {
-            // bsdtar has a bug and hangs, so we are doing it in two steps.
-            if (Core.IsWindows)
-            {
-                var tempDir = PathUtility.EnsureRandomPath(Path.GetTempPath());
-                var tempTarArchive = Path.Combine(tempDir, "temp-file.tar");
-
-                try
-                {
-                    if (!TarArchive.CreateFromPaths(paths, tempTarArchive, relativeTo))
-                    {
-                        return false;
-                    }
-
-                    var process = new Process
-                    {
-                        StartInfo = new ProcessStartInfo
-                        {
-                            FileName = Path.Combine(_pathToZstd, "zstd.exe"),
-                            Arguments = $"-q \"{tempTarArchive}\" -o \"{destinationArchiveFileName}\""
-                        }
-                    };
-
-                    process.Start();
-                    process.WaitForExit();
-
-                    return process.ExitCode == 0;
-                }
-                catch
-                {
-                    return false;
-                }
-                finally
-                {
-                    Directory.Delete(tempDir, true);
-                }
-            }
-
-            return TarArchive.CreateFromPaths(paths, destinationArchiveFileName, relativeTo, "zstd -q", _pathToZstd);
-        }
-    }
-}