The idx v4 type table supersedes the global type queue: workers no longer
enqueue every Write(Type) into a shared ConcurrentQueue during the freeze,
WriteFiles no longer drains and dedupes millions of entries, and the db file
is no longer produced. Loading old saves still reads SerializedTypes.db via
the legacy v2/v3 path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records reference a per-persistence name table by ushort index instead of a
9-byte tag+hash (33 -> 26 bytes per record), eliminating the per-record
xxHash over Type.FullName on the snapshot thread. The loader resolves each
table name once and indexes records into the constructor array. Legacy
v0-v3 loading is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every ISerializable carried SerializedThread/SerializedPosition/SerializedLength
so WriteSnapshot could gather each entity's bytes from the worker heaps in
dictionary order. The idx records absolute positions, so bin order is free -
the snapshot can be written in worker-heap order instead, and the join inverts:
- Chunks are persistence-homogeneous: SerializeAll declares the owner at each
boundary, publishing the partial chunk on change.
- Workers log segments (owner, slot range, heap start) plus one length per
record as they serialize. Positions are implicit because a worker's writes
are contiguous; identity comes from re-walking the same snapshot slots in
the same order (guaranteed stable - mutations divert to the pending queues
until PostWorldSave), or from an entities log on the fallback path.
- WriteSnapshot routes segments by owner, emits idx entries during the
re-walk, and writes each segment's heap bytes as a single span instead of
one copy per entity, which also speeds up the background write phase.
- Persistence self-payloads keep placement as three private fields on the
~dozens of persistence instances; PushSingle is now typed accordingly.
Net effect: 9 bytes (plus padding) of resident state removed from every item,
mobile, guild, and account on every shard; three interface-property stores
per entity leave the drain hot path (stamping dirtied one cache line per
entity mid-freeze - the lengths log is a single sequential stream); and the
vestigial loader-side length stamp is gone. The transient cost is ~4 bytes
per entity in pooled per-worker logs that are released after each write.
The save format is unchanged (idx v3, same loader); only the write-side
mechanics moved. Adds an end-to-end round-trip test that drives real workers
through the chunk source, snapshots from the segment logs, and reloads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured (10M entities through the real chunk source, both monomorphic and
16-subclass polymorphic populations): a bare `foreach { PushToCache(entity); }`
runs at 2.3ns/entity while the same loop carrying the heavy-entity check
(interface SerializedLength read + branch) runs at 5.3-5.7ns - 2.3x slower.
Type diversity barely matters; the cost is the fatter loop body, confirming
that per-entity logic in the push loop defeats the JIT's tight-loop codegen.
Entities over 1MB are rare in practice - realistically only whole
GenericPersistence self-payloads, which are already published as dedicated
single chunks - so the fallback loop drops the check and rare thick entities
ride inside shared chunks (bounded tail, same behavior as the slot-range fast
path). The now-unused HeavyEntityThreshold constant is removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier-1 disasm of a generated-style Serialize showed PGO's guarded
devirtualization already inlines every IGenericWriter.Write body (interface
vs concrete-typed callsites measured identical), so no API or generator
changes are needed. What remained as a real call per primitive write was the
Index property setter - too large to inline due to its range-check throw path,
plus per-write high-water tracking, an AsSpan bounds check, and a
BinaryPrimitives span check.
The write path now reserves capacity once (the existing grow-on-Flush check),
then does an unaligned store through a ref with a raw index increment - the
capacity check proves the store in-bounds, and the index only moves forward
between Seeks. The _bytesWritten high-water mark (used only by
SeekOrigin.End) folds at Seek/Resize instead of per write, which is
equivalent because writes are monotonic between seeks. The Index property
keeps its validating semantics for Seek and subclasses.
BufferWriter also gains class-level implementations of the hottest
IGenericWriter default interface methods (WriteEncodedInt, DateTime,
TimeSpan, Point2D/3D, Rectangle2D/3D, Map, Race): a DIM dispatches again on
`this` for every nested Write even at a devirtualized callsite, and the class
overloads keep the whole write inlined.
Strings (arbitrary UTF-16) previously walked every string twice
(GetByteCount then GetBytes) because the variable-width length prefix
precedes the bytes. Strings of 85 chars or fewer (any content, incl.
surrogate pairs - 85 * 3 = 255 bytes max) now encode once into a 256-byte
stack scratch, then write the prefix and copy. Byte output is identical.
Measured: 34.4 -> 15.7 ns/entity on a generated-style write mix (~20 writes,
58 bytes), 22.5 -> 11.8 ns per short-string write, and the end-to-end freeze
benchmark (10M entities / 1.7GB / 24 cores, dense profile) drops from ~99ms
to ~74-82ms. Combined with the earlier pipeline commits: ~740ms -> ~78ms.
New tests pin byte-level output and position semantics: primitive
little-endian layouts, Seek(End) high-water behavior, growth preservation,
span writes across growth, encoded-int formats, and string equivalence
across the scratch/two-pass boundary with mixed-width UTF-16 content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removes the per-entity handoff from the freeze entirely. GenericEntityPersistence
publishes 4096-slot ranges over its dictionary's backing entries array, and workers
serialize occupied slots (value != null) directly via a ShadowEntry<TValue> struct
that mirrors the runtime's private Dictionary Entry layout. Safe because the
dictionary is frozen during Saving (mutations divert to the pending safety queues).
The layout is proven at startup before any code reads through it: validation
measures the true Entry stride via precise allocation accounting (so all shadow
reads are guaranteed in-bounds), then compares every key and value of a churned,
resized, freelist-exercised dictionary reading value slots as raw pointer bits
only - never materializing a managed reference until the layout is proven. If a
future runtime changes Dictionary internals, validation fails and saves fall back
to the enumerate-and-push path with a logged warning.
The main thread now joins the drain through an inline (threadless) worker after
publishing work, instead of idling while the thread workers finish - worth a full
worker share on the freeze and proportionally more on low-core hosts.
Measured through the real pipeline classes (24 cores, 10M entities, 1.7GB, dense
2-byte write profile): publish cost drops from ~55ms to ~0.1ms, the freeze is now
bound by pure serialize throughput at ~99ms steady state (vs ~740ms before this
branch, ~7.5x), and the first save after boot drops from ~468ms to ~139ms because
fine-grained ranges self-balance without needing size estimates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the per-entity round-robin ConcurrentQueue handoff between the game
loop and the serialization workers with pooled 4096-entity chunks published to
a single shared queue. The producer's per-entity cost drops from a synchronized
enqueue to a plain array store, and workers pulling whole chunks load-balance
dynamically: a worker busy with a thick entity simply takes fewer chunks.
Scheduling changes so indivisible multi-megabyte payloads no longer extend the
freeze tail:
- Persistence.SerializeAll pushes systems largest-first (LPT), using the
previous save's SerializedLength (or the loaded file size on first save).
- Persistence self-payloads and entities whose previous size exceeds 1MB are
published as dedicated single-entity chunks so they spread across workers
instead of riding inside one shared chunk.
- Entity SerializedLength is stamped from the index at load so estimates exist
on the first save after boot.
Worker heaps are pre-sized from the loaded save's .bin sizes (25% headroom) so
the first save doesn't pay copy-on-grow inside the freeze, and the drain loop
uses SpinWait backoff (never Sleep(1)) instead of hammering the queue head
while the producer works. Snapshot writing uses a 1MB FileStream buffer, and
per-worker entity/byte counts are logged at Debug for balance diagnostics.
Measured on a 24-core machine with a synthetic 10M-entity, 1.7GB world
(64/64/32MB system payloads, 24x 2MB thick entities) through the real
pipeline classes: freeze window 740ms -> 122-160ms steady state (~5-6x),
first save 490ms -> 330ms, steady-state allocations converge to zero, and
worker byte loads converge (previous max/min spread ~2x -> ~1.15x for the
small-entity stream).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
## Problem
On headless Linux deployments (systemd service, Docker without a TTY, `nohup`), the ModernUO process pegs a full CPU core even when idle. It does not reproduce on Windows because that runs with an interactive console.
## Root cause
`ConsoleInputHandler` runs a background thread (named "Console Input Handler") that loops on `Console.ReadLine()`. When stdin is **not** an interactive terminal, `Console.ReadLine()` returns `null` at end-of-stream **immediately** on every call, so the loop `continue`s in a tight spin — one core at 100%.
Reproduced in a container running the actual distribution: the "Console Input Handler" thread sat at ~90% CPU on a headless boot; with a blocking stdin it dropped to idle.
## Fix
1. **Detect headless once at startup:** `Core.Headless = Console.IsInputRedirected`.
2. **Extract a testable `ConsoleInputPump`** that owns the input stream: per line read, it *atomically* (under one lock) either delivers the line to a waiting prompt or dispatches a console command, and it **ends on EOF instead of spinning**. Cleanup runs unconditionally in a `finally`, so a pending prompt is always released (never hangs). Replaces the old `async void` loop and the fragile `_expectUserInput` / two-`AutoResetEvent` / `_input` handshake.
3. **`ConsoleInputHandler` becomes a thin headless-aware facade** over the pump. Headless: the reader thread never starts (`Console input disabled (headless: stdin is not a TTY).`), and `ReadLine()` throws a fatal `HeadlessConsoleInputException`.
4. **Data-gating and first-boot prompts** (deserialization "delete bad types? y/n", save-conflict, config/expansion setup) now route through `ConsoleInputHandler.ReadLine()`, so a headless server crashes fatal with a clear message instead of reading `null` (previously an NRE or a silent wrong branch).
Design decision (model b): headless servers are expected to be supplied with configuration/save data (including the owner account); interactive prompts when headless are fatal by design.
## Testing
- New `ConsoleInputPumpTests` (5 tests): EOF ends the loop without spinning; command dispatch; a pending prompt receives the next line; EOF while a prompt is pending completes it with `null` (no hang); a throwing command lookup does not hang a pending prompt. The tests synchronize on real pump state (no `Thread.Sleep`), so they are deterministic on slow CI.
- Full `Server.Tests`: no new failures introduced.
## End-to-end verification (Docker, real distribution)
| | Console Input Handler thread | Container CPU |
|---|---|---|
| Before fix (headless boot) | ~90% | ~199% (2 cores) |
| After fix (headless boot) | **not started** | **~11%** |
After the fix, a headless boot logs `Console input disabled (headless: stdin is not a TTY).`, loads the world normally, and idles instead of spinning.
## Problem
Running the full `UOContent.Tests` suite, the test host **hangs ~2.5 minutes at shutdown and then crashes** (`Test host process crashed` / run aborted). The tests themselves are fine — they complete in ~1s — but the process can't exit.
Captured via `--blame-hang` dump. The blocking thread:
```
System.Threading.WaitHandle.WaitOne()
Server.SerializationThreadWorker.Sleep() SerializationThreadWorker.cs:54 (_stopEvent.WaitOne())
Server.SerializationThreadWorker.Exit() SerializationThreadWorker.cs:61
Server.World.ExitSerializationThreads() World.cs:429
Server.Tests.UOContentFixture..ctor()
```
### Root cause
Both collection fixtures (`UOContentFixture` and `PathfindingTestFixture`) each run the **full process-global ModernUO bootstrap**. `World.Load()` is guarded to run once per process, so the **second** fixture's `World.Load()` is a no-op and does **not** respawn the serialization workers — but `World.ExitSerializationThreads()` is **not** guarded, so the second fixture calls `Exit()` on workers whose threads have already terminated. `Exit()` → `Sleep()` → `_stopEvent.WaitOne()` then blocks forever (a dead thread never sets the event). The first collection's tests run; the second collection's fixture deadlocks in its constructor; the host eventually gets killed.
This is why single-collection (filtered) runs were fine — only one fixture ever bootstraps — but the full suite hangs. It's not a parallelization race: even strictly sequential, the second fixture deadlocks.
## Fix
**(a) Engine — idempotent `SerializationThreadWorker.Exit()`**
A second `Exit()` is now a safe no-op instead of a permanent block. Only the owning (main) thread calls `Exit()`, so no synchronization is needed, and the single-call production shutdown path is unchanged.
**(b) Tests — one shared bootstrap, strictly sequential collections**
- New `TestServerBootstrap.EnsureInitialized()` runs the superset global init **exactly once per process** (lock + once-flag).
- `UOContentFixture` / `PathfindingTestFixture` slim down to delegate to it and no longer tear down global state (which the single-bootstrap model owns for the host's lifetime).
- `[assembly: CollectionBehavior(DisableTestParallelization = true)]` so collections never overlap.
## Result
| | Before | After |
|---|---|---|
| Tests run (full suite) | 258 (UOContent collection deadlocked) | **418** |
| Outcome | 2.5-min hang → host crash | **418 passed, clean exit** |
| Wall time | killed | **~7s** |
## Summary
- Refactors the poison system to separate `Index` (globally unique ID) from `Level` (tier within a family), enabling multiple poison families (Standard, Darkglow, Parasitic) to coexist without collisions
- Implements Darkglow and Parasitic poison special effects from Mondain's Legacy: Darkglow boosts damage by 10% when attacker is ranged, Parasitic heals the attacker for damage dealt in melee range
- Fixes several bugs: `Register()` crashing on duplicate `Level` values across families, `IncreaseLevel()` crossing family boundaries, `InfectiousStrike` and `NinjaWeapons` stripping poison family via level-based lookups, and `ArchCure`/`CleansingWinds` using raw `Level + 1` instead of `IncreaseLevel()`
## Changes
**`Projects/Server/Poison.cs`** — Adds `PoisonFamily` enum and abstract `Family` property. Adds `Index` as unique identifier. Fixes `Register()` to check `Index` uniqueness (not `Level`) and validate the new poison's name (not the existing one's). Fixes `IncreaseLevel()` to use `Index + 1`, naturally respecting family boundaries via Index gaps. Replaces linear name lookup with `Dictionary`-based `PoisonsByName`.
**`Projects/UOContent/Misc/Poison.cs`** — Adds `family` parameter to `PoisonImpl`. Implements Darkglow effect (10% damage boost when `From` >1 tile, cliloc 1072850) and Parasitic effect (heals `From` for damage dealt within 1 tile, cliloc 1060203) in `PoisonTimer.OnTick()`. Renames `m_` fields to `_` convention.
**`Projects/UOContent/Misc/PoisonKinds.cs`** — New file. Moves poison registration out of `PoisonImpl` into `PoisonKinds.Configure()`. Adds `PoisonFamily` to Darkglow/Parasitic registrations. Provides extension properties (`Lesser`, `Deadly`, `LesserDarkglow`, etc.), `GetPoison(int level)` (standard-only), `GetPoisonByFamilyAndLevel()`, and `IsDarkglow`/`IsParasitic` instance helpers.
**`Projects/UOContent/Items/Weapons/Abilities/InfectiousStrike.cs`** — Family-aware poison scaling: Darkglow caps at Deadly (Poisoning/33.3), Parasitic caps at Lethal (Poisoning/25), Standard unchanged. Level bump uses `IncreaseLevel()` with family boundary check.
**`Projects/UOContent/Items/Skill Items/Ninjitsu/NinjaWeapons.cs`** — EvilOmen level bump uses `Poison.IncreaseLevel()` instead of `Poison.GetPoison(Level + 1)`.
**`Projects/UOContent/Spells/Fourth/ArchCure.cs`** and **`CleansingWindsSpell.cs`** — Replace `poison.Level + 1` with `Poison.IncreaseLevel(poison).Level` for family-safe cure chance calculation.
**`Projects/Server/Serialization/SerializationExtensions.cs`** — Serializes/deserializes `Index` instead of `Level`.
**`DarkglowPotion.cs`** / **`ParasiticPotion.cs`** — Point to actual Darkglow/Parasitic poisons instead of placeholder `Greater`.
**`PotionKeg.cs`** / **`BasePotion.cs`** — Adds Darkglow, Parasitic, Invisibility, and FlintsPungentBrew to `PotionEffect` enum and keg label support.
## Test plan
- [ ] `dotnet build` compiles cleanly (verified, 0 warnings 0 errors)
- [ ] Verify `PoisonKinds.Configure()` registers all poisons without throwing (Register bug fix)
- [ ] Standard poison behavior unchanged — PoisonField, PoisonSpell, SerpentArrow, SavageShaman, TrappableContainer all use `GetPoison(int level)` which now correctly filters to Standard family
- [ ] Darkglow: poison tick deals +10% damage when attacker is >1 tile away, sends "Darkglow poison increases your damage!" message
- [ ] Parasitic: poison tick heals attacker for damage dealt when within 1 tile, sends heal message
- [ ] InfectiousStrike preserves poison family and respects family-specific skill scaling
- [ ] EvilOmen + NinjaWeapons level bump stays within poison family
- [ ] ArchCure/CleansingWinds cure chance calculations work correctly across all poison families
- [ ] Serialization round-trips correctly using Index
Fixes serialization/deserialization edge cases with BitArray. If you use BitArray, you will need to migrate.
1. Change the type in the migration JSON file (if there is one) from `BitArray` to `byte[]` for all the versions you need to migrate.
2. Then in the `MigrateFrom`, use the following function to convert the field from a byte[] back to the BitArray.
Example Migration JSON:
```json
{
"name": "RestrictedSpells",
"type": "byte[]",
"rule": "ArrayMigrationRule",
"ruleArguments": [
"byte",
"PrimitiveTypeMigrationRule",
""
]
},
```
Migration function to use in MigrateFrom:
```cs
public static BitArray MigrateBitArray(byte[] data, int bitLength) => new(data) { Length = bitLength };
```
Example use:
```cs
private void MigrateFrom(V0Content content)
{
// ... deserialize
_restrictedSpells = content.RestrictedSpells.MigrateBitArray(SpellRegistry.Types.Length);
_restrictedSkills = content.RestrictedSkills.MigrateBitArray(SkillInfo.Table.Length);
// ... rest of deserialize
}
```
### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.
> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
### Summary
- Fixes infinite loop with binary file writer
- Removes extra buffer copying with binary file writer
- Removes storing type counts during world save file writing
- Fixes display cache self-deletion warning during world load
### Summary
* Fixes spawner timer deserialization
* Adds a check for a null timer and allows the timer to get recreated
* Adds PotionKeg reverse lookup
* Heavily optimizes decimal serialize/deserialize
### Summary
The BitArray class will be optimized over the next several years for various platforms/hardware and maintaining a duplicate for serialization is not practical. Removing the custom implementation. Recommend against using BitArray for serialization unless it is absolutely necessary.
### Summary
- Fixes usernames not being `Intern`ed
- Reverts methods related to getting accounts from returning `Account` to `IAccount`.
- Makes `IAccount` also `ISerializable`
- Adds `IGenericReader.ReadAccount()` and `IGenericWriter.Write(IAccount)` -> The read method supports the original serialization of username, and using `IAccount.Serial`. The write method only serializes the `Serial`.
- Exposes `ReadStringRaw()` to allow some advanced scenarios.
### Summary
- [X] Fixed a bug where entity persistence was serialized out of order, causing world corruption.
- [X] Fixed LastSerialized not being utilized properly and dangling references still becoming an issue.
- [X] Added a new `GenericEntityPersistence<T>` type to encapsulate `ISerializable` serialization.
- [X] Removing the custom logic and moved Items, Mobiles, Guilds, and Accounts to GenericEntityPersistence.
- [X] Changed serialization to use the singleton pattern to reduce calling methods from stored variables.
### Summary
Adds a generic entity persistence. This can be used to create new entity types that have a `Serial`.
Here is an example:
```cs
public class BOBEntries : GenericEntitySerialization<IBOBEntry>
{
public static void Configure()
{
Configure("BOBEntries");
}
}
```
The annotation tells the system what folder to serialize the entries to. The class/interface (`IBOBEntry`) is the root type that implements `ISerializable`.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
## **MAJOR CHANGE**
* `Stabled` has been moved to `PlayerMobile`.
* New methods added, `PlayerMobile.AddStabled` and `PlayerMobile.RemoveStabled`.
* Added `PlayerMobile.AddFollower` and `PlayerMobile.RemoveFollower`.
* `Stabled`, `AutoStabled`, and `AllFollowers` are now `HashSet` and **_CAN BE NULL_**.
### Summary
* Fixes monster abilities causing harm to the monster through reflect
* Adds `CanTriggerAgainstSelf` to override this for healing or some other self-affecting ability
* Fixes a major memory leak where `UnsummonTimer` from animated dead spell lasts up-to 24hrs and therefore holds onto references of dead/deleted mobs.
* Fixes another minor leak where a mob is not unregistered from the animated dead spell list until the next spell cast.
### Summary
Updates BufferWriter with more standard ways of writing primitives. Eliminates looping to write a string per @jaedan's suggestion.
Note: This change assumes we don't have crazy large strings.
### BREAKING CHANGE ###
The constructor for `TextDefinition` has been removed. Instead use `TextDefinition.Of()` or cast the integer/string to TextDefinition.
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
