Commit graph

3 commits

Author SHA1 Message Date
Kamron Batman
9a25054809
test: remove trivial, duplicate and content-value tests (#2721)
## Summary
Many tests proved a specific change when it landed but don't need to run on every CI build forever. This removes **202 test methods and 9 files: 313 test cases, 2,247 → 1,934**. UOContent.Tests drops from about 14s to about 5s locally.

Removed:
- **Tautologies.** Asserting a constant, a constructor readback or a trivial getter, or testing BCL behavior.
- **Exact duplicates** of a surviving test on the same code path.
- **Obsolete regressions** whose failure can no longer be expressed.
- **Game-content values** shard owners customize (era drop rates, loot and skill tables, single-creature tuning). Each customization would otherwise force a test fix.

## How it was decided
1. Every test was classified, with a reason, by a per-shard sweep over all 287 test files.
2. Each proposed deletion was then checked by a second, adversarial reviewer against the surviving tests and the production code.
3. Anything that was the only coverage of a separate code path was **kept**, even where it looked like a duplicate. Examples: the unsigned `SpanReader`/`SpanWriter` methods, the at-location versus bounds enumerators, null-map enumerables, the security filters, archive-journal state transitions. About 130 proposed deletions were rejected this way.

Whole files removed:
- `FishingTests`, apart from 3 tests: the map-edge scan, serpent cleanup, and SOS chest consumption. The SOS test now stops at the first chest instead of measuring a catch rate. The multi-million-cast rate simulations are gone.
- `LordOaksTests` and `MeerEnragedTests` (single-creature tests).
- `AutoArchiveHelperTests`, `JsonDiscoverableTypeAttributeTests`, `DecimalSerializationTests` and `EnumConversionTests` (tautologies, or tests of their own local helpers).
- `NetworkStatsTests`.
- `MigratedDataLoadTests` (a subset of `AllSpawnFilesLoadTests`).
- `SpawnerFixtureCapture`: a capture tool, not a test, and its `version == 12` assert can no longer pass. The `.bin` fixtures it produced stay.

## Tests
- Server.Tests: 811 pass.
- UOContent.Tests: 1,123 pass.

A follow-up PR, stacked on this one, fixes how the remaining tests behave: shared state, CI-vacuous tests, harness duplication.
2026-10-18 20:08:19 -07:00
Kamron Batman
078be52777
fix: reject archive entries that extract outside the output directory (#2720)
## Summary
`ManagedArchive.ExtractTarFromStream` joined each tar entry name directly onto the output directory, so an entry such as `../x`, `sub/../../x` or an absolute path was written outside it. The restore path (`AutoArchive` → `ExtractTarZstd`/`ExtractTarGz`/`ExtractTar`) feeds archives through this method.

Each entry now resolves to a full path and must stay under the output root (case-insensitive on Windows). An entry that escapes fails the extraction; the existing `catch` in each `Extract*` method logs it and returns `false`.

The input is the server's own backup archives, so this is hardening against a tampered or foreign archive, not something a player can reach.

## Tests
- `ExtractTar_EntryOutsideOutputDirectory_IsRejected` (`../escaped.txt`, `sub/../../escaped.txt`) and `ExtractTar_RootedEntry_IsRejected`. All three fail without the fix.
- All 18 `ManagedArchiveTests` pass.
2026-10-18 20:06:23 -07:00
Kamron Batman
3b4e1137f6
feat: Implements new robust/pluggable backup/archive system. (#2388)
## Summary

Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.

### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics

### What changed

**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.

**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).

**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.

**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).

**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.

**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.

### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule

### Files

| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |

### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup

### Plugin example (external repo)
```csharp
public static class S3Plugin
{
    public static void Configure()
    {
        ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
    }
}
```

## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach
2026-03-22 19:51:20 -07:00