The idx v4 type table supersedes the global type queue: workers no longer
enqueue every Write(Type) into a shared ConcurrentQueue during the freeze,
WriteFiles no longer drains and dedupes millions of entries, and the db file
is no longer produced. Loading old saves still reads SerializedTypes.db via
the legacy v2/v3 path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every ISerializable carried SerializedThread/SerializedPosition/SerializedLength
so WriteSnapshot could gather each entity's bytes from the worker heaps in
dictionary order. The idx records absolute positions, so bin order is free -
the snapshot can be written in worker-heap order instead, and the join inverts:
- Chunks are persistence-homogeneous: SerializeAll declares the owner at each
boundary, publishing the partial chunk on change.
- Workers log segments (owner, slot range, heap start) plus one length per
record as they serialize. Positions are implicit because a worker's writes
are contiguous; identity comes from re-walking the same snapshot slots in
the same order (guaranteed stable - mutations divert to the pending queues
until PostWorldSave), or from an entities log on the fallback path.
- WriteSnapshot routes segments by owner, emits idx entries during the
re-walk, and writes each segment's heap bytes as a single span instead of
one copy per entity, which also speeds up the background write phase.
- Persistence self-payloads keep placement as three private fields on the
~dozens of persistence instances; PushSingle is now typed accordingly.
Net effect: 9 bytes (plus padding) of resident state removed from every item,
mobile, guild, and account on every shard; three interface-property stores
per entity leave the drain hot path (stamping dirtied one cache line per
entity mid-freeze - the lengths log is a single sequential stream); and the
vestigial loader-side length stamp is gone. The transient cost is ~4 bytes
per entity in pooled per-worker logs that are released after each write.
The save format is unchanged (idx v3, same loader); only the write-side
mechanics moved. Adds an end-to-end round-trip test that drives real workers
through the chunk source, snapshots from the segment logs, and reloads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the per-entity round-robin ConcurrentQueue handoff between the game
loop and the serialization workers with pooled 4096-entity chunks published to
a single shared queue. The producer's per-entity cost drops from a synchronized
enqueue to a plain array store, and workers pulling whole chunks load-balance
dynamically: a worker busy with a thick entity simply takes fewer chunks.
Scheduling changes so indivisible multi-megabyte payloads no longer extend the
freeze tail:
- Persistence.SerializeAll pushes systems largest-first (LPT), using the
previous save's SerializedLength (or the loaded file size on first save).
- Persistence self-payloads and entities whose previous size exceeds 1MB are
published as dedicated single-entity chunks so they spread across workers
instead of riding inside one shared chunk.
- Entity SerializedLength is stamped from the index at load so estimates exist
on the first save after boot.
Worker heaps are pre-sized from the loaded save's .bin sizes (25% headroom) so
the first save doesn't pay copy-on-grow inside the freeze, and the drain loop
uses SpinWait backoff (never Sleep(1)) instead of hammering the queue head
while the producer works. Snapshot writing uses a 1MB FileStream buffer, and
per-worker entity/byte counts are logged at Debug for balance diagnostics.
Measured on a 24-core machine with a synthetic 10M-entity, 1.7GB world
(64/64/32MB system payloads, 24x 2MB thick entities) through the real
pipeline classes: freeze window 740ms -> 122-160ms steady state (~5-6x),
first save 490ms -> 330ms, steady-state allocations converge to zero, and
worker byte loads converge (previous max/min spread ~2x -> ~1.15x for the
small-entity stream).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.
> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
### Summary
- [X] Fixed a bug where entity persistence was serialized out of order, causing world corruption.
- [X] Fixed LastSerialized not being utilized properly and dangling references still becoming an issue.
- [X] Added a new `GenericEntityPersistence<T>` type to encapsulate `ISerializable` serialization.
- [X] Removing the custom logic and moved Items, Mobiles, Guilds, and Accounts to GenericEntityPersistence.
- [X] Changed serialization to use the singleton pattern to reduce calling methods from stored variables.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
- [X] Caches DateTime.NowUtc on the game loop (not other threads)
- [X] Replaces all locations where it makes sense
- [X] Adds Min/Max for `IComparable` (TimeSpan, DateTimes, etc)
Closes#261
