Commit graph

112 commits

Author SHA1 Message Date
Kamron Batman
3b4e1137f6
feat: Implements new robust/pluggable backup/archive system. (#2388)
## Summary

Overhauls the backup/archive system to address stability, robustness, fault tolerance, performance, and pluggability.

### Problems solved
- **Critical bug**: `Interlocked.CompareExchange` arguments were reversed — the concurrent archive guard never actually prevented concurrent archives
- **Brittle compression**: Relied on spawning external `zstd.exe`/`tar.exe`/`bsdtar.exe` processes with platform-specific workarounds (Windows two-step hack, auto-downloading bsdtar from libarchive.org)
- **No fault tolerance**: Partial archives possible if process killed mid-write; no recovery mechanism; originals deleted before verification
- **No extensibility**: No way to add remote backup destinations (S3, rsync) without modifying core code
- **Silent failures**: Bare `catch` blocks swallowed exceptions with no diagnostics

### What changed

**Cross-platform streaming compression** — Replaced external process spawning with `System.Formats.Tar` (built-in .NET) + `ZstdNet` (native libzstd P/Invoke). Single-pass streaming: `TarWriter → CompressionStream → FileStream`. No intermediate `.tar` file, no platform-specific code paths.

**Archive journal** — JSON-based operation journal (`Archives/.archive-journal.json`) tracks state machine: `Started → Archived → Distributed → Completed` (or `Failed`). On startup, recovers interrupted operations (cleans temp files, completes pruning).

**Verify before delete** — Archives are verified (entry count check) before deleting source backups. Temp-file-then-atomic-rename pattern prevents partial archives.

**Retry logic** — File operations (moves, deletes) retry with linear backoff for transient I/O failures (antivirus locks, file copy operations).

**Plugin architecture** — `IArchiveDestination` interface in `Projects/Server/` enables external plugins (loaded via `assemblies.json`) to receive completed archives. `ArchiveDestinationRegistry` tracks destinations. Conservative pruning: source backups preserved if any destination with retention fails.

**Configurable** — Retention counts (hourly/daily/monthly), compression level, retry settings, backup max age all configurable via `ServerConfiguration`.

### New admin commands
- `[ArchiveStatus` — Shows journal state, destinations, next scheduled archive times
- `[ArchiveNow` — Forces immediate rollup regardless of schedule

### Files

| Change | File |
|--------|------|
| New | `Server/Saves/ArchiveEnums.cs`, `IArchiveDestination.cs`, `ArchiveDestinationRegistry.cs`, `ArchiveEventArgs.cs` |
| New | `Server/Events/ArchiveEvents.cs` (partial EventSink) |
| New | `UOContent/Compression/ManagedArchive.cs` |
| New | `UOContent/World Saves/ArchiveJournal.cs`, `LocalArchiveDestination.cs` |
| Rewrite | `UOContent/World Saves/AutoArchive.cs` |
| Modified | `UOContent/World Saves/SaveCommands.cs`, `Server/Utilities/PathUtility.cs`, `UOContent.csproj` |
| Deleted | `UOContent/Compression/TarArchive.cs`, `ZstdArchive.cs` |
| Tests | 29 new tests (ManagedArchive, ArchiveJournal, AutoArchiveHelpers) |

### Backward compatibility
- Existing `.tar.zst` archives are fully readable by the new managed code
- Config keys preserved; new keys use sensible defaults
- `autoArchive.compressionFormat` setting removed (Zstd only)
- Legacy `bsdtar/` directory logged as removable on startup

### Plugin example (external repo)
```csharp
public static class S3Plugin
{
    public static void Configure()
    {
        ArchiveDestinationRegistry.Register(new S3Destination("my-bucket", "us-east-1"));
    }
}
```

## Test plan
- [x] 29 new unit tests covering archive create/extract/count, journal state machine, recovery, destinations
- [x] All 969 tests pass (671 Server + 298 UOContent)
- [x] Manual: run server, trigger save, verify backup created and archive rolled up
- [x] Manual: kill server mid-archive, restart, verify journal recovery
- [x] Manual: test with large save files (~500MB+) to benchmark streaming vs old approach
2026-03-22 19:51:20 -07:00
Kamron Batman
e1e1a7c640
fix: Bumps deps. Updates copyrights (#2353) 2026-03-05 19:36:54 -08:00
Joe
e77a566f32
feat: Implements passive Detect Hidden mechanics (#2342) 2026-02-28 11:24:49 -08:00
Kamron Batman
fd2f90cba9
feat: Adds SerializableFieldChanged option (#2324) 2026-02-01 21:28:26 -08:00
Kamron Batman
bde072f81c
feat: Bumps Serialization Generator for better support (#2292)
### Summary

* Adds better `SaveFlag` support.
* Fixes `SortedSet` support.
* Adds custom comparer support for `SortedSet`.
* Fixes various bugs.
* Adds support for `struct`, `record`, and `generic` classes.
* Adds support for `readonly` fields by skipping serializaton entirely.
* Adds support for interface fields that inherit ISerializable
* Adds tests.

See: 59a92cc49a
2025-12-26 14:27:24 -08:00
Kamron Batman
3a3f5ee518
feat: Adds .NET 10 / C# 14 support. (#2258)
### Summary
* Adds .NET 10 support
* Bumps to C# 14
2025-11-11 11:26:34 -08:00
Kamron Batman
2df62fba5a
chore: Bumps various dependencies (#2237) 2025-07-22 16:04:04 -07:00
Kamron Batman
61d074213b
chore(deps): Updates MailKit to 4.11.0 and Microsoft deps to 9.0.4 (#2156) 2025-04-12 12:54:40 -07:00
Kamron Batman
41e909cd22
fix: Fixes serialization generator issue with arrays (#2120) 2025-02-12 22:30:46 -08:00
Kamron Batman
af027cdac6
Bumps dependencies. Fixes bugs with serialization generator (#2119) 2025-02-12 20:50:28 -08:00
Kamron Batman
d92b735c18
chore(deps): Bumps Nerdbank, Hashing, FileSystemGlobbing, xunit, and C# version (#2077) 2025-01-18 17:02:18 -08:00
Guyute
9d0d454b4e
feat: Moves logger to a separate assembly for reuse (#2001) 2024-12-30 20:25:34 -08:00
Kamron Batman
60b97f80d8
chore(deps): Bumps xunit to 3.0, serialization generator to 2.12.18, high performance to 8.4, and mailkit to 4.9 (#2032) 2024-12-27 12:56:41 -08:00
Reetus
f33e218c0b
fix: Fix HolidayTree not serializing components (#2024) 2024-12-17 08:59:50 -08:00
Kamron Batman
c75514cc04
feat: Updates to .NET 9 (#1984)
### Summary

* Bumps to .NET 9 with updated dependencies
* Comparing a value type against null is no longer allowed
* CI/CD now uses the version specified in global.json
* Serialization generator updated to .NET 9 with bug fixes, fixes to turkish language, and parallelization
2024-12-08 10:16:34 -08:00
Kamron Batman
26c7f5c49b
chore(deps): Bumps xunit to 2.9.2, Serilog to 4.0.2, and MailKit to 4.0.8 (#1973) 2024-10-15 17:04:33 -07:00
Kamron Batman
4e2fcee7a2
chore(deps): Bumps xunit to 2.9.1 and CommunityToolkit.HighPerformance to 8.3.2 (#1962) 2024-09-25 10:27:37 -07:00
Kamron Batman
4110556e89
chore(deps): Bumps dependencies (#1944) 2024-09-05 21:24:47 -07:00
Guyute
95cd8749a3
feat: Adds PlayerDeathEvent and CreatureDeathEvent using code generated events (#1927)
### Summary
- Adds [Code Generated Events](https://github.com/modernuo/CodeGeneratedEvents)
- Removes EventSink.PlayerDeath
- Adds `PlayerDeathEvent` and `CreatureDeathEvent` using code generated events

> [!Important]
> **Developer Note**
> Use `[OnEvent(nameof(PlayerMobile.PlayerDeathEvent))]` instead of `EventSink.PlayerDeath` delegate
> Check this commit for examples of how to use this.
2024-09-05 21:23:03 -07:00
dependabot[bot]
bb24b330e1
chore(deps): Bump MailKit from 4.7.1 to 4.7.1.1 (#1872) 2024-07-20 18:04:18 -07:00
Kamron Batman
db31c07017
fix: Updates xunit to 2.9, runner to 2.8.2, and mailkit to 4.7.1 (#1871) 2024-07-12 19:59:33 -07:00
Kamron Batman
c914d3d722
chore: Bumps MailKit to 4.7.0 & Cleans up bad code formatting (#1857) 2024-07-04 19:55:14 -07:00
Kamron Batman
8ec203f387
feat: Updates serialization to use MMF (considerable memory savings) (#1841)
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.

> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
2024-06-23 10:51:20 -07:00
Kamron Batman
5f8f7982eb
fix: Fixes publishing (#1808)
### Summary
- Fixes publishing issues when the Application project was created.
- Fixes missing dotnet clean file deletions
2024-05-31 12:59:59 -07:00
Kamron Batman
60a3a6f501
feat: Splits Server in Core and Application (#1806)
> [!Note]
> **Developer Note**
> Now developers will only need to build/run the Application project instead of everything.
> When adding new projects, make sure to: 
> 1. Add a reference to that project in the Application project.
> 2. Add the dll file to the Distribution/Data/assemblies.json file

### Summary
- Adds an application project
- Consolidates process restarts to use `Core.Kill(true)`
- Removes some old messaging, for example processor optimization
- Fixes missing build cleanup
2024-05-30 23:34:03 -07:00
Kamron Batman
21c5e46565
fix: Fixes libdeflate gc pin issue (#1794) 2024-05-25 18:11:46 -07:00
Stefano Merotta
50d3779d9b
fix: Adds dispose call to LibDeflateBinding on process exits (#1784) 2024-05-21 13:40:03 -07:00
dependabot[bot]
8054b29e5c
chore(deps): Bump MailKit from 4.5.0 to 4.6.0 (#1775)
Bumps [MailKit](https://github.com/jstedfast/MailKit) from 4.5.0 to 4.6.0.
- [Changelog](https://github.com/jstedfast/MailKit/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/jstedfast/MailKit/commits)

---
updated-dependencies:
- dependency-name: MailKit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-18 10:24:40 -07:00
Kamron Batman
1f701e7b55
feat: Replaces Zlib with LibDeflate (#1774)
> [!Warning]
> Users on Linux/OSX will need to follow the Readme
> and make sure `libdeflate` is properly installed

> [!Note]
> **Developer Note**
> The API for compression has changed. Use `Deflate.Standard` for the same functionality.

### Summary
* Replaces Zlib with LibDeflate for a 50% performance improvement!
* Adds MacOS 14 to properly test Arm64
2024-05-16 22:53:01 -07:00
dependabot[bot]
72261a6b9c
chore(deps): Bump MailKit from 4.4.0 to 4.5.0 (#1732)
Bumps [MailKit](https://github.com/jstedfast/MailKit) from 4.4.0 to 4.5.0.
- [Changelog](https://github.com/jstedfast/MailKit/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/jstedfast/MailKit/compare/4.4.0...4.5.0)

---
updated-dependencies:
- dependency-name: MailKit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-15 18:31:16 -07:00
dependabot[bot]
50fcee12dc
chore(deps): Bump MailKit from 4.3.0 to 4.4.0 (#1694)
Bumps [MailKit](https://github.com/jstedfast/MailKit) from 4.3.0 to 4.4.0.
- [Changelog](https://github.com/jstedfast/MailKit/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/jstedfast/MailKit/compare/4.3.0...4.4.0)

---
updated-dependencies:
- dependency-name: MailKit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-04 21:10:10 -08:00
Kamron Batman
61e9dd29ac
fix: Adds missing migration files. (#1642)
### Summary

- Bumps serialization generator. v2.10.9 fixes an issue with nuget publishing where files were missing.
- Fixes camps missing serialization of their prisoner.
- Adds missing serialization migration files.
2023-12-20 13:02:01 -08:00
Kamron Batman
b7882df136
fix: Fixes serialization and dirty tracking conveniences (#1627)
### Summary
- Fixes issues with non-_ISerializable_ objects having bad serialization (Skill/Stat. Mods)
- Fixes issues with MarkDirty and namespaces: https://github.com/modernuo/SerializationGenerator/issues/29
- Fixes missing dirty tracking for some data structures.
- Fixes cascading MarkDirty for non-serializable types that have owners that are also non-serializable types.

### New Codegenned API
When a data structure (Array, List, Dictionary, HashSet, etc) is source generated for serialization, new methods are added which will handle dirty tracking. Use these instead of the built in methods for that data structure.

_All Data Structures_
```cs
public void Clear<PropertyName>();
```

_Lists and Sets_
```cs
public void AddTo<PropertyName>(V value);
public void RemoveFrom<PropertyName>(V value);
```

_Lists_
```cs
public void InsertInto<PropertyName>(int index, V value);
public void RemoveFrom<PropertyName>At(int index);
```

_Dictionaries_
```cs
public void AddTo<PropertyName>(T key, V value);
public void RemoveFrom<PropertyName>(T key);
public void ReplaceIn<PropertyName>(T key, V value);
```

Over time, they will be cleaned up and more variants added such as `bool RemoveFrom<PropertyName>(T key, out V value)`
2023-12-03 15:55:31 -08:00
Kamron Batman
e21ff0eb28
fix: Codegens BaseMulti, Container, and VirtualCheck (#1624) 2023-12-02 10:00:02 -08:00
Kamron Batman
d93e363017
fix: Fixes VS support by reverting source generators back to net standard 2.0 (#1609) 2023-11-21 17:10:13 -08:00
Kamron Batman
0da6e46d67
feat: Updates to .NET Standard 2.1 source generators (#1602) 2023-11-19 10:03:28 -08:00
Kamron Batman
2e4668dbe5
feat: Updates to .NET 8. (#1542)
### Breaking Changes
- Updating to .NET 8 - Required O/S's have slightly changed.
2023-11-14 17:08:09 -08:00
dependabot[bot]
bd79509400
chore(deps): Bump MailKit from 4.2.0 to 4.3.0 (#1590) 2023-11-13 22:11:41 -08:00
dependabot[bot]
01b864f955
chore(deps): Bump MailKit from 4.1.0 to 4.2.0 (#1493)
Bumps [MailKit](https://github.com/jstedfast/MailKit) from 4.1.0 to 4.2.0.
- [Changelog](https://github.com/jstedfast/MailKit/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/jstedfast/MailKit/compare/4.1.0...4.2.0)

---
updated-dependencies:
- dependency-name: MailKit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 22:37:30 -07:00
Kamron Batman
ffb0b24e1a
fix: Codegens talismans - Fixes memory leak (#1469) 2023-08-23 22:41:48 -07:00
Kamron Batman
455b11de9c
fix: Updates schema generator to 2.6.4 (#1452) 2023-08-11 00:09:56 -07:00
Kamron Batman
09fc30546b
fix: Fixes RunebookEntry serialization and removes unused migrations (#1449)
### Summary

- [X] Removes migration files were committed that aren't actually (and have never) been used.
- [X] Moves runebook entry from being manually serialized to using the serialization generator.
2023-08-10 19:27:01 -07:00
Kamron Batman
d9d04f1f38
fix: Adds better arm64 support (#1409) 2023-07-26 20:43:42 -07:00
Kamron Batman
94199f1186
fix: Adds migration checks to CICD, updates dependencies, adds Fedora 37, alpine 3.17 support (#1309)
- [X] Automatically collapses migration files in github PRs.
- [X] CI/CD now checks to see if migrations are missing or modified.
- [X] Updates dependencies
- [X] Adds Fedora 37, Alpine 3.17 support
2022-12-22 12:12:47 -08:00
Kamron Batman
32c15f214b
feat: Adds GUID support for codegen (#1251) 2022-11-14 21:25:51 -08:00
Kamron Batman
936000ecf7
feat: Adds SerializedCommandProperty (#1249)
## Breaking Change
* Removes `SerializableFieldAttrAttribute` in favor of `SerializedPropertyAttr`.

## Important Change
* Adds `SerializedCommandProperty`

Example:
```cs
    [InvalidateProperties]
    [SerializableField(0)]
    [SerializedCommandProperty(AccessLevel.GameMaster)]
    private Mobile _completedBy;
```
2022-11-14 18:24:33 -08:00
Kamron Batman
e9c6faab33
fix: Updates dependencies for .NET 7 (#1240) 2022-11-11 11:54:29 -08:00
Kamron Batman
810061db8c
chore: Update from .NET 7 preview to release. (#1234) 2022-11-08 09:46:48 -08:00
Kamron Batman
5bd41d5b68
feat: Adds .NET 7 & Arm64 support (#1229)
## BREAKING CHANGE
**Publishing for linux will no longer include runtimes**

## Major Changes
* Adds .NET 7 support.
* Adds ARM64 support (experimental).
* Changes linux support to be open-ended against anything .NET 7 supports.
* Fixes native library resolutions. (Make sure to install `dev` versions of the libraries)
* Updates README
* Adds Ubuntu 22, CentOS 8/9 Stream to CI/CD.

## TODOs:
* Update modernuo.com docs

Closes #1173
Closes #1159
2022-11-06 09:59:11 -08:00
Kamron Batman
8a26afe28b
fix: Fixes regression in primitive UO codegen (#1219) 2022-10-30 01:58:45 -07:00