Removes the per-entity handoff from the freeze entirely. GenericEntityPersistence
publishes 4096-slot ranges over its dictionary's backing entries array, and workers
serialize occupied slots (value != null) directly via a ShadowEntry<TValue> struct
that mirrors the runtime's private Dictionary Entry layout. Safe because the
dictionary is frozen during Saving (mutations divert to the pending safety queues).
The layout is proven at startup before any code reads through it: validation
measures the true Entry stride via precise allocation accounting (so all shadow
reads are guaranteed in-bounds), then compares every key and value of a churned,
resized, freelist-exercised dictionary reading value slots as raw pointer bits
only - never materializing a managed reference until the layout is proven. If a
future runtime changes Dictionary internals, validation fails and saves fall back
to the enumerate-and-push path with a logged warning.
The main thread now joins the drain through an inline (threadless) worker after
publishing work, instead of idling while the thread workers finish - worth a full
worker share on the freeze and proportionally more on low-core hosts.
Measured through the real pipeline classes (24 cores, 10M entities, 1.7GB, dense
2-byte write profile): publish cost drops from ~55ms to ~0.1ms, the freeze is now
bound by pure serialize throughput at ~99ms steady state (vs ~740ms before this
branch, ~7.5x), and the first save after boot drops from ~468ms to ~139ms because
fine-grained ranges self-balance without needing size estimates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the per-entity round-robin ConcurrentQueue handoff between the game
loop and the serialization workers with pooled 4096-entity chunks published to
a single shared queue. The producer's per-entity cost drops from a synchronized
enqueue to a plain array store, and workers pulling whole chunks load-balance
dynamically: a worker busy with a thick entity simply takes fewer chunks.
Scheduling changes so indivisible multi-megabyte payloads no longer extend the
freeze tail:
- Persistence.SerializeAll pushes systems largest-first (LPT), using the
previous save's SerializedLength (or the loaded file size on first save).
- Persistence self-payloads and entities whose previous size exceeds 1MB are
published as dedicated single-entity chunks so they spread across workers
instead of riding inside one shared chunk.
- Entity SerializedLength is stamped from the index at load so estimates exist
on the first save after boot.
Worker heaps are pre-sized from the loaded save's .bin sizes (25% headroom) so
the first save doesn't pay copy-on-grow inside the freeze, and the drain loop
uses SpinWait backoff (never Sleep(1)) instead of hammering the queue head
while the producer works. Snapshot writing uses a 1MB FileStream buffer, and
per-worker entity/byte counts are logged at Debug for balance diagnostics.
Measured on a 24-core machine with a synthetic 10M-entity, 1.7GB world
(64/64/32MB system payloads, 24x 2MB thick entities) through the real
pipeline classes: freeze window 740ms -> 122-160ms steady state (~5-6x),
first save 490ms -> 330ms, steady-state allocations converge to zero, and
worker byte loads converge (previous max/min spread ~2x -> ~1.15x for the
small-entity stream).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
## Problem
On headless Linux deployments (systemd service, Docker without a TTY, `nohup`), the ModernUO process pegs a full CPU core even when idle. It does not reproduce on Windows because that runs with an interactive console.
## Root cause
`ConsoleInputHandler` runs a background thread (named "Console Input Handler") that loops on `Console.ReadLine()`. When stdin is **not** an interactive terminal, `Console.ReadLine()` returns `null` at end-of-stream **immediately** on every call, so the loop `continue`s in a tight spin — one core at 100%.
Reproduced in a container running the actual distribution: the "Console Input Handler" thread sat at ~90% CPU on a headless boot; with a blocking stdin it dropped to idle.
## Fix
1. **Detect headless once at startup:** `Core.Headless = Console.IsInputRedirected`.
2. **Extract a testable `ConsoleInputPump`** that owns the input stream: per line read, it *atomically* (under one lock) either delivers the line to a waiting prompt or dispatches a console command, and it **ends on EOF instead of spinning**. Cleanup runs unconditionally in a `finally`, so a pending prompt is always released (never hangs). Replaces the old `async void` loop and the fragile `_expectUserInput` / two-`AutoResetEvent` / `_input` handshake.
3. **`ConsoleInputHandler` becomes a thin headless-aware facade** over the pump. Headless: the reader thread never starts (`Console input disabled (headless: stdin is not a TTY).`), and `ReadLine()` throws a fatal `HeadlessConsoleInputException`.
4. **Data-gating and first-boot prompts** (deserialization "delete bad types? y/n", save-conflict, config/expansion setup) now route through `ConsoleInputHandler.ReadLine()`, so a headless server crashes fatal with a clear message instead of reading `null` (previously an NRE or a silent wrong branch).
Design decision (model b): headless servers are expected to be supplied with configuration/save data (including the owner account); interactive prompts when headless are fatal by design.
## Testing
- New `ConsoleInputPumpTests` (5 tests): EOF ends the loop without spinning; command dispatch; a pending prompt receives the next line; EOF while a prompt is pending completes it with `null` (no hang); a throwing command lookup does not hang a pending prompt. The tests synchronize on real pump state (no `Thread.Sleep`), so they are deterministic on slow CI.
- Full `Server.Tests`: no new failures introduced.
## End-to-end verification (Docker, real distribution)
| | Console Input Handler thread | Container CPU |
|---|---|---|
| Before fix (headless boot) | ~90% | ~199% (2 cores) |
| After fix (headless boot) | **not started** | **~11%** |
After the fix, a headless boot logs `Console input disabled (headless: stdin is not a TTY).`, loads the world normally, and idles instead of spinning.
## Problem
Running the full `UOContent.Tests` suite, the test host **hangs ~2.5 minutes at shutdown and then crashes** (`Test host process crashed` / run aborted). The tests themselves are fine — they complete in ~1s — but the process can't exit.
Captured via `--blame-hang` dump. The blocking thread:
```
System.Threading.WaitHandle.WaitOne()
Server.SerializationThreadWorker.Sleep() SerializationThreadWorker.cs:54 (_stopEvent.WaitOne())
Server.SerializationThreadWorker.Exit() SerializationThreadWorker.cs:61
Server.World.ExitSerializationThreads() World.cs:429
Server.Tests.UOContentFixture..ctor()
```
### Root cause
Both collection fixtures (`UOContentFixture` and `PathfindingTestFixture`) each run the **full process-global ModernUO bootstrap**. `World.Load()` is guarded to run once per process, so the **second** fixture's `World.Load()` is a no-op and does **not** respawn the serialization workers — but `World.ExitSerializationThreads()` is **not** guarded, so the second fixture calls `Exit()` on workers whose threads have already terminated. `Exit()` → `Sleep()` → `_stopEvent.WaitOne()` then blocks forever (a dead thread never sets the event). The first collection's tests run; the second collection's fixture deadlocks in its constructor; the host eventually gets killed.
This is why single-collection (filtered) runs were fine — only one fixture ever bootstraps — but the full suite hangs. It's not a parallelization race: even strictly sequential, the second fixture deadlocks.
## Fix
**(a) Engine — idempotent `SerializationThreadWorker.Exit()`**
A second `Exit()` is now a safe no-op instead of a permanent block. Only the owning (main) thread calls `Exit()`, so no synchronization is needed, and the single-call production shutdown path is unchanged.
**(b) Tests — one shared bootstrap, strictly sequential collections**
- New `TestServerBootstrap.EnsureInitialized()` runs the superset global init **exactly once per process** (lock + once-flag).
- `UOContentFixture` / `PathfindingTestFixture` slim down to delegate to it and no longer tear down global state (which the single-bootstrap model owns for the host's lifetime).
- `[assembly: CollectionBehavior(DisableTestParallelization = true)]` so collections never overlap.
## Result
| | Before | After |
|---|---|---|
| Tests run (full suite) | 258 (UOContent collection deadlocked) | **418** |
| Outcome | 2.5-min hang → host crash | **418 passed, clean exit** |
| Wall time | killed | **~7s** |
## Summary
- Refactors the poison system to separate `Index` (globally unique ID) from `Level` (tier within a family), enabling multiple poison families (Standard, Darkglow, Parasitic) to coexist without collisions
- Implements Darkglow and Parasitic poison special effects from Mondain's Legacy: Darkglow boosts damage by 10% when attacker is ranged, Parasitic heals the attacker for damage dealt in melee range
- Fixes several bugs: `Register()` crashing on duplicate `Level` values across families, `IncreaseLevel()` crossing family boundaries, `InfectiousStrike` and `NinjaWeapons` stripping poison family via level-based lookups, and `ArchCure`/`CleansingWinds` using raw `Level + 1` instead of `IncreaseLevel()`
## Changes
**`Projects/Server/Poison.cs`** — Adds `PoisonFamily` enum and abstract `Family` property. Adds `Index` as unique identifier. Fixes `Register()` to check `Index` uniqueness (not `Level`) and validate the new poison's name (not the existing one's). Fixes `IncreaseLevel()` to use `Index + 1`, naturally respecting family boundaries via Index gaps. Replaces linear name lookup with `Dictionary`-based `PoisonsByName`.
**`Projects/UOContent/Misc/Poison.cs`** — Adds `family` parameter to `PoisonImpl`. Implements Darkglow effect (10% damage boost when `From` >1 tile, cliloc 1072850) and Parasitic effect (heals `From` for damage dealt within 1 tile, cliloc 1060203) in `PoisonTimer.OnTick()`. Renames `m_` fields to `_` convention.
**`Projects/UOContent/Misc/PoisonKinds.cs`** — New file. Moves poison registration out of `PoisonImpl` into `PoisonKinds.Configure()`. Adds `PoisonFamily` to Darkglow/Parasitic registrations. Provides extension properties (`Lesser`, `Deadly`, `LesserDarkglow`, etc.), `GetPoison(int level)` (standard-only), `GetPoisonByFamilyAndLevel()`, and `IsDarkglow`/`IsParasitic` instance helpers.
**`Projects/UOContent/Items/Weapons/Abilities/InfectiousStrike.cs`** — Family-aware poison scaling: Darkglow caps at Deadly (Poisoning/33.3), Parasitic caps at Lethal (Poisoning/25), Standard unchanged. Level bump uses `IncreaseLevel()` with family boundary check.
**`Projects/UOContent/Items/Skill Items/Ninjitsu/NinjaWeapons.cs`** — EvilOmen level bump uses `Poison.IncreaseLevel()` instead of `Poison.GetPoison(Level + 1)`.
**`Projects/UOContent/Spells/Fourth/ArchCure.cs`** and **`CleansingWindsSpell.cs`** — Replace `poison.Level + 1` with `Poison.IncreaseLevel(poison).Level` for family-safe cure chance calculation.
**`Projects/Server/Serialization/SerializationExtensions.cs`** — Serializes/deserializes `Index` instead of `Level`.
**`DarkglowPotion.cs`** / **`ParasiticPotion.cs`** — Point to actual Darkglow/Parasitic poisons instead of placeholder `Greater`.
**`PotionKeg.cs`** / **`BasePotion.cs`** — Adds Darkglow, Parasitic, Invisibility, and FlintsPungentBrew to `PotionEffect` enum and keg label support.
## Test plan
- [ ] `dotnet build` compiles cleanly (verified, 0 warnings 0 errors)
- [ ] Verify `PoisonKinds.Configure()` registers all poisons without throwing (Register bug fix)
- [ ] Standard poison behavior unchanged — PoisonField, PoisonSpell, SerpentArrow, SavageShaman, TrappableContainer all use `GetPoison(int level)` which now correctly filters to Standard family
- [ ] Darkglow: poison tick deals +10% damage when attacker is >1 tile away, sends "Darkglow poison increases your damage!" message
- [ ] Parasitic: poison tick heals attacker for damage dealt when within 1 tile, sends heal message
- [ ] InfectiousStrike preserves poison family and respects family-specific skill scaling
- [ ] EvilOmen + NinjaWeapons level bump stays within poison family
- [ ] ArchCure/CleansingWinds cure chance calculations work correctly across all poison families
- [ ] Serialization round-trips correctly using Index
Fixes serialization/deserialization edge cases with BitArray. If you use BitArray, you will need to migrate.
1. Change the type in the migration JSON file (if there is one) from `BitArray` to `byte[]` for all the versions you need to migrate.
2. Then in the `MigrateFrom`, use the following function to convert the field from a byte[] back to the BitArray.
Example Migration JSON:
```json
{
"name": "RestrictedSpells",
"type": "byte[]",
"rule": "ArrayMigrationRule",
"ruleArguments": [
"byte",
"PrimitiveTypeMigrationRule",
""
]
},
```
Migration function to use in MigrateFrom:
```cs
public static BitArray MigrateBitArray(byte[] data, int bitLength) => new(data) { Length = bitLength };
```
Example use:
```cs
private void MigrateFrom(V0Content content)
{
// ... deserialize
_restrictedSpells = content.RestrictedSpells.MigrateBitArray(SpellRegistry.Types.Length);
_restrictedSkills = content.RestrictedSkills.MigrateBitArray(SkillInfo.Table.Length);
// ... rest of deserialize
}
```
### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.
> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
### Summary
- Fixes infinite loop with binary file writer
- Removes extra buffer copying with binary file writer
- Removes storing type counts during world save file writing
- Fixes display cache self-deletion warning during world load
### Summary
* Fixes spawner timer deserialization
* Adds a check for a null timer and allows the timer to get recreated
* Adds PotionKeg reverse lookup
* Heavily optimizes decimal serialize/deserialize
### Summary
The BitArray class will be optimized over the next several years for various platforms/hardware and maintaining a duplicate for serialization is not practical. Removing the custom implementation. Recommend against using BitArray for serialization unless it is absolutely necessary.
### Summary
- Fixes usernames not being `Intern`ed
- Reverts methods related to getting accounts from returning `Account` to `IAccount`.
- Makes `IAccount` also `ISerializable`
- Adds `IGenericReader.ReadAccount()` and `IGenericWriter.Write(IAccount)` -> The read method supports the original serialization of username, and using `IAccount.Serial`. The write method only serializes the `Serial`.
- Exposes `ReadStringRaw()` to allow some advanced scenarios.
### Summary
- [X] Fixed a bug where entity persistence was serialized out of order, causing world corruption.
- [X] Fixed LastSerialized not being utilized properly and dangling references still becoming an issue.
- [X] Added a new `GenericEntityPersistence<T>` type to encapsulate `ISerializable` serialization.
- [X] Removing the custom logic and moved Items, Mobiles, Guilds, and Accounts to GenericEntityPersistence.
- [X] Changed serialization to use the singleton pattern to reduce calling methods from stored variables.
### Summary
Adds a generic entity persistence. This can be used to create new entity types that have a `Serial`.
Here is an example:
```cs
public class BOBEntries : GenericEntitySerialization<IBOBEntry>
{
public static void Configure()
{
Configure("BOBEntries");
}
}
```
The annotation tells the system what folder to serialize the entries to. The class/interface (`IBOBEntry`) is the root type that implements `ISerializable`.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
## **MAJOR CHANGE**
* `Stabled` has been moved to `PlayerMobile`.
* New methods added, `PlayerMobile.AddStabled` and `PlayerMobile.RemoveStabled`.
* Added `PlayerMobile.AddFollower` and `PlayerMobile.RemoveFollower`.
* `Stabled`, `AutoStabled`, and `AllFollowers` are now `HashSet` and **_CAN BE NULL_**.
### Summary
* Fixes monster abilities causing harm to the monster through reflect
* Adds `CanTriggerAgainstSelf` to override this for healing or some other self-affecting ability
* Fixes a major memory leak where `UnsummonTimer` from animated dead spell lasts up-to 24hrs and therefore holds onto references of dead/deleted mobs.
* Fixes another minor leak where a mob is not unregistered from the animated dead spell list until the next spell cast.
### Summary
Updates BufferWriter with more standard ways of writing primitives. Eliminates looping to write a string per @jaedan's suggestion.
Note: This change assumes we don't have crazy large strings.
### BREAKING CHANGE ###
The constructor for `TextDefinition` has been removed. Instead use `TextDefinition.Of()` or cast the integer/string to TextDefinition.
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
Fixes bit array serialization. This may cause objects that were serialized by bit array to fail to deserialize. I am sorry, please accept my condolences. It is probably easiest to just delete those objects. If it becomes a major problem, contact me and I'll help with a hacky per-case solution.
* Adds a custom BitArray class with the following added features:
* ctor for creating BitArray against read only span
* ctor for creating BitArray against BinaryReader
* CopyTo to copy a BitArray to a Span
* Adds BitArray to UO Primitive serialization so it can be codegenned.
