3b4e1137f6
## Summary
Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.
### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics
### What changed
**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.
**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).
**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.
**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).
**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.
**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.
### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule
### Files
| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |
### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup
### Plugin example (external repo)
```csharp
public static class S3Plugin
{
public static void Configure()
{
ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
}
}
```
## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach
58 lines
4.1 KiB
XML
58 lines
4.1 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current">
|
|
<PropertyGroup>
|
|
<RootNamespace>Server</RootNamespace>
|
|
<AssemblyName>UOContent</AssemblyName>
|
|
<Product>ModernUO Content</Product>
|
|
<OutDir>..\..\Distribution\Assemblies</OutDir>
|
|
<PublishDir>..\..\Distribution\Assemblies</PublishDir>
|
|
<Configurations>Debug;Release;Analyze</Configurations>
|
|
</PropertyGroup>
|
|
<Target Name="CleanPub" AfterTargets="Clean">
|
|
<Message Text="UOContent: Removing distribution assemblies..." />
|
|
<Delete Files="..\..\Distribution\Assemblies\Argon2.Bindings.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\libargon2.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\BouncyCastle.Cryptography.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\MailKit.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\MimeKit.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\Microsoft.Toolkit.HighPerformance.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\CommunityToolkit.HighPerformance.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\Microsoft.Extensions.FileSystemGlobbing.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\Serilog.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\Serilog.Sinks.Async.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\Serilog.Sinks.Console.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\System.Security.Cryptography.Pkcs.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\ref\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\$(AssemblyName).deps.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\$(AssemblyName).pdb" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\LibDeflate.Bindings.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\libdeflate.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\libdeflate.dylib" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\ZstdNet.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Assemblies\ModernUO.Serialization.Annotations.dll" ContinueOnError="true" />
|
|
</Target>
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\Logger\Logger.csproj" />
|
|
<ProjectReference Include="..\Server\Server.csproj" Private="false" PrivateAssets="All" IncludeAssets="None">
|
|
<IncludeInPackage>false</IncludeInPackage>
|
|
</ProjectReference>
|
|
<PackageReference Include="LibDeflate.Bindings" Version="1.0.3" />
|
|
<PackageReference Include="MailKit" Version="4.15.1" />
|
|
<PackageReference Include="Microsoft.Extensions.FileSystemGlobbing" Version="10.0.3" />
|
|
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.0" />
|
|
<PackageReference Include="Argon2.Bindings" Version="1.17.0" />
|
|
<PackageReference Include="ModernUO.CodeGeneratedEvents.Annotations" Version="1.0.0" />
|
|
<PackageReference Include="ModernUO.CodeGeneratedEvents.Generator" Version="1.0.3.2" PrivateAssets="all" />
|
|
<PackageReference Include="ZstdNet" Version="1.5.7" />
|
|
|
|
<PackageReference Include="ModernUO.Serialization.Annotations" Version="2.14.2" />
|
|
<PackageReference Include="ModernUO.Serialization.Generator" Version="2.14.2" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<AdditionalFiles Include="Migrations/*.v*.json" />
|
|
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
|
<_Parameter1>UOContent.Tests</_Parameter1>
|
|
</AssemblyAttribute>
|
|
</ItemGroup>
|
|
</Project>
|