Commit graph

20 commits

Author SHA1 Message Date
Kamron Batman
3b4e1137f6
feat: Implements new robust/pluggable backup/archive system. (#2388)
## Summary

Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.

### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics

### What changed

**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.

**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).

**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.

**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).

**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.

**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.

### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule

### Files

| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |

### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup

### Plugin example (external repo)
```csharp
public static class S3Plugin
{
    public static void Configure()
    {
        ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
    }
}
```

## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach
2026-03-22 19:51:20 -07:00
Kamron Batman
da517f56c4
chore: Fixes variable types for possible performance issues (#2172) 2025-05-01 21:54:41 -07:00
Kamron Batman
44df304575
fix: Archive locally is not an async function (#1735) 2024-04-21 08:08:12 -07:00
Kamron Batman
3244704ea2
fix: Fixes directory copying (#1668) 2024-02-09 23:25:20 -08:00
Kamron Batman
331f24cb71
fix: Fixes TimeSpan rounding issues (#1610)
### Summary

TLDR - .NET Framework rounded TimeSpan (and some DateTime) methods to the nearest millisecond. This was actually _expected_ accidentally for parts of RunUO.

We are aiming to fix this and clean up some other bad assumptions.

Closes #1604
2023-11-22 18:03:56 -08:00
Kamron Batman
2e4668dbe5
feat: Updates to .NET 8. (#1542)
### Breaking Changes
- Updating to .NET 8 - Required O/S's have slightly changed.
2023-11-14 17:08:09 -08:00
mdodkins
d366e02251
fix: Fixes missing ordinal string comparisons arguments. (#1431)
---------

Co-authored-by: Kamron Batman <3953314+kamronbatman@users.noreply.github.com>
2023-07-23 20:12:38 -07:00
Kamron Batman
d8cfb6b935
fix: Makes logging more consistent (#1246) 2022-11-13 00:36:23 -08:00
Kamron Batman
03fb36c869
fix: Adds ISpanParsable and fixes command conditionals (#1241)
* Adds `ISpanParsable<T>`
* Removes `[Parsable]`
* Fixes querying by serial, body, and a few others.
* Adds `Parse` to `Rectangle3D`
* Fixes AutoArchive NPE


Closes #1209
2022-11-12 00:26:02 -08:00
Kamron Batman
f268d5d4e2
fix: Cleans up core code (#1187)
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`

Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.

View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
2022-10-10 21:47:08 -07:00
Kamron Batman
0d19a5ff99
fix: Do not use unsafe work queue for auto archiving (#1116) 2022-07-14 11:27:29 -07:00
Kamron Batman
c31bf20d0e
feat: Updates to .NET 6 (#843)
* Fixes an issue with moving directories across volumes
* Removes usages of WebClient
* Removes usages of Cryptographic Providers

Note: Even though .NET 6 introduces Xoshiro RNG, there is no way to control the seed. I'll do some reconciliation of Xoshiro so it functions closer to the built in one. For the most part, it has parity though.
Benchmarks show there is nothing odd about the implementations, they are within 1ns of each other.
2021-11-13 13:38:01 -08:00
Kamron Batman
fe0abc4a8f
fix: Cleans up archiving and fixes wrong relative path used (#825) 2021-10-09 14:36:46 -07:00
Kamron Batman
f3b722b9d6
fix: Fixes epoll calls on linux (#818) 2021-10-05 20:06:00 -07:00
Kamron Batman
a55e271a69
fix: Cleans up file system paths and archiving (#815)
* Makes EnsureDirectory properly work for relative and absolute paths
* Adds a `PathUtility.GetFullPath` which returns full paths for relative paths to `Core.BaseDirectory`. If the path is absolute, it will return as-is.
* Moves EnsureDirectory to `PathUtility`. So `ScriptsHandler.EnsureDirectory` and `AssemblyHandler.EnsureDirectory` are now `PathUtility.EnsureDirectory`
* Fixes crash guard so that it copies accounts properly.
* Changes world save and auto archive to use a random folder name inside of the temp folder.
2021-10-05 08:54:44 -07:00
Kamron Batman
8eaa859332
fix: Fixes code genning armor, clothing, and BODs (#810)
* Fixes code genning clothing
* Fixes code genning armor
* Fixes code genning BODs
2021-09-28 22:09:34 -07:00
Kamron Batman
b4ab052241
fix: Adds more folder safety to auto archiving (#803) 2021-09-26 18:13:52 -07:00
Michael Whelehan
f1a4195543
fix: Ensure backup folder exists before rolling up directories too (#802) 2021-09-26 15:12:33 -07:00
Kamron Batman
b5799c33af
fix: Fixes backup folder not being created (#801) 2021-09-26 09:06:11 -07:00
Kamron Batman
3f6a87483b
feat: Adds auto archiving (#794)
## Adds Auto Archiving
Backups are archived once an hour, day, and month. Archives older than 60 days are pruned automatically.

_Note: Automatic pruning is off by default_

### Archive compression format
The following formats are supported:
* Zstd (The fastest with best compression ratio)
* GZip
* Zip
* None (Tar)

_Note: By default archives use [zstandard](http://facebook.github.io/zstd/) format.
The archive format can be changed in modernuo.json `autoArchive.compressionFormat`_

### Restoring world from archive
Move the archive to the Saves folder (tar.zst file). On startup the server will extract the file and restore the latest save. See pictures below.

### Manually extracting an archives
#### Windows
* Use the latest version of [7-zip w/ ZStandard](https://github.com/mcmilk/7-Zip-zstd/releases/latest)
  1. Extract the `.tar.zst` file.
  2. Extract the `.tar` file. (Yes you have to do it in two steps)
* On Windows 10 you can use the command line. `zstd.exe` is in the Assemblies folder after building ModernUO.
  1. `tar --use-compress-program "Distribution\Assemblies\zstd.exe -d" -xvf "Archives\Hourly\archivefile.tar.zst" -C "path to where you want to extract it"`
#### Mac
* Install [Keka](https://www.keka.io)
  1. Drop the .tar.zst onto the keka interface.
#### Linux
  1. Install zstd from a package manager
  2. Run `tar -I zstd -xvf "Archives\Hourly\archivefile.tar.zst" -C "path to where you want to extract it"`

### Other changes
* Changes Autosave to occur at the same time no matter when the server is booted.
* Adds `[SaveFrequency <delay> [warning]`command to set save frequency and warning frequency in-game.
* Adds support for time zones that are configurable. The system timezone can also be manually configured. Check `TimeZoneHandler.cs` for details.

<img width="257" alt="Screen Shot 2021-09-22 at 11 23 18 PM" src="https://user-images.githubusercontent.com/3953314/134464929-a5bf3cd8-2ef0-4476-a9ad-71d0816a19ac.png">
<img width="241" alt="Screen Shot 2021-09-22 at 11 23 39 PM" src="https://user-images.githubusercontent.com/3953314/134464945-5f6f96dc-3d1e-434d-8256-5c6b3705e786.png">
<img width="836" alt="Screen Shot 2021-09-22 at 11 34 39 PM" src="https://user-images.githubusercontent.com/3953314/134464957-625e57f2-ef47-4ca1-a0a7-cd40c9b6539c.png">
<img width="631" alt="Screen Shot 2021-09-22 at 11 34 50 PM" src="https://user-images.githubusercontent.com/3953314/134464969-b2d65c0d-f8f5-497a-a832-5d805e8e0b22.png">
2021-09-25 17:22:05 -07:00