## Why
The shard owner, on a Starlink CGNAT address, was blocked by the imported reputation blocklist.
The cause was not CrowdSec. The address was a literal line in `ip-blocklist.txt`, so `BlocklistFilter` denied it at accept and then promoted it — and clearing the CrowdSec decision could not fix it either, because the file entry re-reports within `promoteSuppression` of every reconnect attempt.
This is structural, not a one-off. Reputation feeds list shared consumer address space constantly: on CGNAT one public address fronts many subscribers **at the same time**, so a single abusive customer gets the address listed and everyone else behind it is blocked with them. Where leases rotate, a listing says little about whoever holds the address now. Around 1,000 Starlink addresses sit in the current list.
So exemptions go where they cost nothing, and escalation is driven by what a connection actually does.
## Generator — `tools/Export-IpBlocklist.ps1`
`-AllowlistFile` takes multiple paths, subtracted from the merged set before the output is written. Defaults to every `ip-allowlist*.txt` beside the output, merged into one allow set:
- `ip-allowlist.txt` — operator exemptions, created once and **never rewritten**
- `ip-allowlist-<name>.txt` — a carve-out you built, regenerable and copyable between shards
**Subtraction is range-correct.** An allowlisted address inside a blocked CIDR splits that CIDR around the hole rather than being silently ignored. This also fixes `-ExcludeAnonymizers`, which parsed CIDR entries into `$anonCidr` and then only ever subtracted singles.
**No carve-out ships.** A carve-out names a real network, and which ones a shard should exempt depends on where its players actually are — so publishing one would make that policy call for every shard and put a specific provider's address space in the repo. The script builds them on request instead:
```powershell
.\Export-IpBlocklist.ps1 -AddCarveout starlink -Asn 14593
```
Carve-outs are **discovered, not configured**: every `ip-allowlist*.txt` beside the output is subtracted, by the generator and by the shard, so a file an admin adds needs no config edit and no code change. Each carries an `asn=` marker in its header, which is how `-RefreshCarveouts` rebuilds it without the script keeping a list of anyone's networks; a hand-written allowlist has no marker and is never rewritten.
Prefixes come from **announcements, not ownership records**, because registry data disagrees with what is actually routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and `206.83.96.0/19` reads as APNIC in RDAP even though `206.83.96/21` is announced by Starlink.
Editing an allowlist bypasses `-MinInterval`, so a just-added exemption isn't indistinguishable from the allowlist not working. A Starlink carve-out, if you build one, costs **~4,300 IPs + ~144 CIDRs of 4.2M (0.10%)**.
## Allowlists
**`FileAllowlist`** reads the same files the generator subtracts, so an operator entry means "leave this address alone" for real. Subtraction alone only covers being *blocked*; behavioural detections never consult the blocklist, so without this a carve-out was quietly routed around — one scanner behind a shared address was enough to get everyone behind it contributed and firewalled, with nothing in the shard's own config explaining why. Reading the files also means an entry applies on the next reload rather than the next regeneration, which is what matters when someone is complaining now.
**`LoginAllowlist`** is earned by authenticating, with a 90-day TTL because an address that logged in years ago is a stranger. Its own store rather than `Account.LoginIPs`, which has no timestamps and cannot be backfilled. An entry is evidence rather than a licence: 10 suppressed contributions in an hour revokes it, and a fresh login forgives the tally.
Both are consulted **only after the blocklist has already matched**, so a normal accept pays nothing for them and the accept gate stays allowlist-free. `BanExemptions` combines them behind `BanChannel.IsExempt` and suppresses escalation only — every local defence still applies.
Two limits, both deliberate and documented in the class: `LoginAllowlist` **cannot bootstrap** (an entry is only earned by getting in, so it never repairs an existing false positive), and it is weakest on rotating CGNAT. That is why `FileAllowlist` is the fix for those, and why it is manual.
## Behavioural detection
| Reason | Trigger |
|---|---|
| `silent-connect` | Reaped after 5s having sent **zero bytes** |
| `invalid-seed` | Opened with a zero seed |
| `foreign-protocol` | Positively identified as HTTP, TLS or SSH |
**`ForeignProtocol` inverts the test.** Asking "is this a good UO client?" cannot work: `LoginEncryption.ClientDecrypt` is a byte-for-byte stream XOR, so a legitimate client with encryption enabled when the shard expects none sends a structurally perfect connection whose payload is noise. "Speaks HTTP" is safe where "unreadable" is not — however misconfigured a UO client is, it never sends `GET / HTTP/1.1`.
Nothing assumes arrival framing. TCP has no message boundaries, so a rule of the form "these bytes must arrive together" is broken by construction and drops real players on poor links. A prefix match with too few bytes to confirm waits for more. A four-byte seed can legitimately spell `GET ` (the address 71.69.84.32) or `0x16 0x03 0x0?` (22.3.x.x), so confirmation requires the request line to continue in printable ASCII or an actual ClientHello inside a plausible record — a real client's fifth byte is a packet id (`0x80`, `0x91`, `0xEF`), none of them printable, so those collisions fall through.
Everything is keyed on **bytes-received rather than elapsed time**. A connection that sent something and ran out of time is far more likely a slow link than an attack, and banning those produces the worst failure mode available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being firewalled off.
## `AutoDenylist`
A short-lived local hold (15m) on behavioural detections, as `IConnectionFilter` + `IBanReporter` over one store so the engine detection sites never reach into content.
This closes the gap where a flood pays for a socket, buffer and `NetState` slot per connection while waiting for the OS bouncer — the verdicts that matter most are reachable only *after* reading bytes — and it is the entire defence on a shard running no bouncer, which is the default config. Not persisted: a holding pen that survives restarts is a ban without a ban's review.
Cost: one dictionary lookup on a usually-empty dict per accept.
## `BanReasons`
Centralises the reason slugs. `IsBehavioral` is an **opt-in** set, not "everything except manual", so a future reason escalates normally instead of silently inheriting an exemption or entering a local denylist.
This caught a real bug during review: the first cut of the exemption swallowed `manual` admin bans (`Commands.cs`, three sites in `AdminGump`) for any allowlisted address.
## Fixes found in review
- **`BanConfiguration.Settings` was null until `Configure()` ran**, while the reap path dereferences it every `Slice()`. A harness driving `NetState.Slice()` directly hit an NRE that presented as flaky because it depended on whether an earlier test had already called `Configure()` — which is why it failed on some CI platforms and not others. Now starts at the record's defaults, with idempotency tracked by a flag; this also removes the same latent NRE from the pre-existing rate-limit path.
- **`-AllowlistFile` was typed `[string]`** while documented and used as a list, so passing two paths would have collapsed them into one string.
## Layout and docs
Content network code moves out of `Misc/` into `UOContent/Network/`, one concern per folder — `AutoDenylist/`, `Blocklist/`, `CrowdSec/`, `Firewall/`, `LoginAllowlist/`, `Packets/`. **Namespaces are untouched**, so these are pure file moves (git tracks all 16 as renames).
`dev-docs/ip-bans-and-allowlists.md` documents the subsystem, leading with the operator process for unblocking a player — including the three things that look sufficient and are not: deleting the CrowdSec decision alone, editing `ip-blocklist.txt` by hand, and `cscli allowlists` alone. `.gitignore` covers the new config files.
## Testing
Build clean. **Server.Tests 810 passed**, **UOContent.Tests 637 passed**, zero warnings. This branch adds 38 tests; the rest of the delta is main's, since this is rebased on current `main`.
New coverage: TTL boundary and renewal, private-address exclusion, manual-ban-never-exempt, unopted-reason-never-exempt, strike revocation, quiet-window reset, login forgiveness, file-allowlist CIDR coverage, file-allowlist not spending the earned list's strikes, denylist expiry-on-read, cap enforcement, lapsed-entry reclaim, HTTP/TLS/SSH identification, seed-collision fall-through, and encrypted-login-is-not-foreign.
Generator verified end-to-end against live feeds: a clean run ships no carve-out, `-AddCarveout starlink -Asn 14593` fetches and collapses 213 prefixes to 115 ranges in 0.1s over 4.2M entries, `-RefreshCarveouts` rediscovers it by its `asn=` marker, a hand-written allowlist is left untouched, and deleting a carve-out drops it rather than having it rewritten. CIDR splitting verified exhaustively: a single-IP hole in a /24 leaves exactly 255 of 256 addresses blocked.
## Operator note
Existing installs are unaffected until the generator next runs, which creates `ip-allowlist.txt` and nothing else. To unblock someone: add the address to that file and delete any live CrowdSec decision — the existing ban outlives the config change. The shard picks the entry up on its next reload, so re-running the generator is optional.
A shard whose players are on CGNAT (satellite, mobile, or an ISP short on IPv4) will likely also want `-AddCarveout`; see `dev-docs/ip-bans-and-allowlists.md`.
## Also included: a latent CI failure this PR surfaced
`fix(tests): serialize test classes that rent through STArrayPool` touches a property-list test file that has nothing to do with this feature. It is here because it was failing macOS CI, and it is trivially cherry-pickable out if you would rather it went to `main` on its own — **which may be the better call, since it is failing `main` today.**
CI has since gone green with it applied.
`STArrayPool` is single-threaded by design and its bucket cache is a plain `static`, not `[ThreadStatic]`, with a check-then-act initialize in `Return()`:
```csharp
var cacheBuckets = _cacheBuckets ?? InitializeBuckets();
```
Two threads both see null, both initialize, and the loser trips `Debug.Assert(_cacheBuckets is null)`. Anything renting from it has to stay off parallel test threads — which is what the `DisableParallelization` collections are for.
- `ObjectPropertyListReentrancyTests` and `ObjectPropertyListNestedBuildTests` (added in #2555) build property lists, which rent the interpolation buffer, but were not in the sequential collection — unlike `PropertyListInvalidationDuringBuildTests` in the same file. This is a **latent failure already on `main`**; it is timing-dependent, so it shows on some platforms and not others.
- `AutoDenylistTests` (added here) has the same exposure: its cap tests reach `AutoDenylist.Sweep`, which rents a `PooledRefList` without `mt`. The blocklist tests need no marking because `BlocklistSnapshot.Build` asks for the `mt` pool explicitly.
No production change — `STArrayPool` is the right pool on the game loop, where both `Sweep` and the property list actually run.
## Deliberately not included
Waiting for a fragmented four-byte seed at `AwaitingSeed`. It looked like a bug but the disconnect is a deliberate defence: only pre-0xEF clients reach it (0xEF goes through `HandlePacket`, which already waits for its 21 bytes), and waiting converts an instant drop into a full 5s slot hold for a client sending one or two bytes, or a loris dribbling a byte every few seconds. Against a fixed 4096-entry `MaxConnections` table that trades capacity that matters for a fragmentation case a reconnect already fixes.
Reduces the world-save freeze window from ~740ms to ~78ms (measured on a synthetic 10M-entity / 1.7GB world, 24 cores, through the real pipeline classes) by removing the per-entity handoff between the game loop and the serialization workers, fixing how large indivisible payloads are scheduled, rewriting the BufferWriter hot path, removing per-entity placement state entirely, and finally replacing the global serialized-types tracking with a per-file type table (idx v4) that also shrinks idx files by ~21% and speeds the background write phase. The pipeline has also been validated end-to-end on live-copy worlds in the multi-million-entity range, where the freeze is drain-bound (real `Serialize()` costs far more CPU per byte than synthetic writes) — the same structural wins hold, and entity/file round-trips are byte-clean across both load paths.
## The problem
The freeze window is `max(main-thread handoff, slowest worker drain)`:
1. **The producer was the bottleneck.** The main thread round-robined every entity through per-worker `ConcurrentQueue`s — two interlocked ops per entity, ~740ms of freeze floor at 10M entities before any serialization happened.
2. **Round-robin distributes count, not cost.** "Deep" systems (50MB generic persistence blobs) and "thick" entities (100K-item storage keys) landed on arbitrary workers, producing lopsided drain times on large worlds.
3. **Worst-case scheduling.** `GenericEntityPersistence.Serialize` pushed its self-payload *after* all entities, and generic persistences sort last in the registry — so the biggest indivisible blobs started serializing at the very end, extending the freeze by their entire duration.
## The fix
**Commit 1 — chunked handoff + LPT scheduling + heap pre-sizing:**
- Pooled 4096-entity chunks published to one shared queue; workers pull chunks and load-balance dynamically (a worker busy with a thick entity simply takes fewer chunks).
- `Persistence.SerializeAll` pushes systems largest-first (LPT) using the previous save's payload size (or loaded file size on first boot); self-payloads get dedicated single-entity chunks so they overlap the entity stream instead of ending it.
- Worker heaps pre-size from the loaded save's `.bin` totals, eliminating copy-on-grow inside the first save's freeze.
- `SpinWait` backoff in the drain loop (never `Sleep(1)`), per-worker balance stats logged in debug builds (the call site is compiled out of Release), 1MB snapshot write buffer.
**Commit 2 — workers iterate the dictionaries directly + main thread joins the drain:**
- `GenericEntityPersistence` publishes 4096-slot ranges over its dictionary's backing entries array; workers serialize occupied slots (`value != null`) directly through a `ShadowEntry<TValue>` struct mirroring the runtime's private `Entry` layout. Safe because the dictionary is frozen during `Saving` (mutations divert to the pending safety queues).
- The layout is **proven at startup before any code reads through it**: validation measures the true `Entry` stride via precise allocation accounting (guaranteeing all shadow reads are in-bounds), then verifies every key/value of a churned, resized, freelist-exercised dictionary — reading value slots as raw pointer bits only, never materializing a managed reference until the layout is proven. If a future runtime changes `Dictionary` internals, validation fails with a logged warning and saves fall back to the (fully maintained) enumerate-and-push path.
- The main thread joins the drain via an inline worker after publishing, instead of idling — worth a full worker share, proportionally more on low-core hosts.
**Commit 3 — 2.2x faster BufferWriter write path, single-pass short strings:**
- PGO already devirtualizes and inlines every `IGenericWriter.Write` callsite (interface vs concrete measured identical) — the real per-write cost was the non-inlinable `Index` setter (range-check throw path + per-write high-water tracking) plus span bounds checks. Writes now reserve capacity once, then do an unaligned store through a ref with a raw index increment; the high-water mark folds at Seek/Resize instead of per write.
- Class-level implementations of the hottest default interface methods keep nested writes inlined (a DIM re-dispatches on `this` even at a devirtualized callsite).
- Strings of 85 chars or fewer encode once into a stack scratch instead of walking the string twice (`GetByteCount` + `GetBytes`). Byte output is identical.
- Measured: 34.4 → 15.7 ns/entity on a generated-style write mix; end-to-end freeze ~99ms → ~74-82ms.
**Commit 4 — branch-free fallback push loop:**
- A bare `foreach { PushToCache(entity); }` runs at 2.3ns/entity; the same loop carrying a per-entity heavy-check runs 2.3x slower — the cost is the fatter loop body defeating tight-loop codegen. Entity-level >1MB payloads are rare enough to ride in shared chunks; system self-payloads (the large ones) are still explicitly scheduled largest-first.
**Commit 5 — drop the 9-byte per-entity placement state; snapshots write from worker segment logs:**
- Every `ISerializable` carried `SerializedThread/SerializedPosition/SerializedLength` so `WriteSnapshot` could gather each entity's bytes from the worker heaps in dictionary order. But the idx records absolute positions — bin order is free — so the snapshot is now written in worker-heap order and the join inverts: workers log segments (owner, slot range, heap start) plus one length per record as they serialize; positions are implicit because a worker's writes are contiguous, and identity comes from re-walking the same snapshot slots in the same order (stable until `PostWorldSave`).
- Chunks are persistence-homogeneous (the partial chunk publishes at each `SerializeAll` boundary) so segments route to files by owner with zero per-entity state. Self-payloads keep placement as three private fields on the handful of persistence instances.
- Net: 9 bytes (plus padding) of resident state removed from every item, mobile, guild, and account on every shard; three interface-property stores per entity leave the drain hot path (stamping dirtied one cache line per entity mid-freeze — the lengths log is one sequential stream); each segment's bytes hit the bin as a single span write instead of one copy per entity, speeding the background write phase; and `IGenericSerializable` shrinks to just `Serialize(IGenericWriter)`. Transient cost: ~4 bytes per entity in pooled per-worker logs, released after each write. The save format was unchanged at this point (idx v3, same loader); the v4 bump comes later in the branch.
**Commit 6 — staged file writes replace memory-mapped snapshot writing:**
- `MemoryMapFileWriter` is removed. `FileBufferWriter` composes through the full `BufferWriter` raw write path into a pooled staging block that drains to the file as large sequential positional writes (`RandomAccess.Write`); seeks flush the block and move the file offset, so backwards patches (the idx entity count) become small positional writes.
- Memory-mapped composition paid a soft page fault on every composed page plus unpredictable dirty-section teardown stalls at dispose — measured ~4x slower end-to-end than staged writes at snapshot sizes.
**Commits 7–11 — idx v4: per-file type table replaces SerializedTypes.db and all runtime type tracking:**
- Previously every `Write(Type)` from every worker enqueued into a shared `ConcurrentQueue<Type>` during the freeze (interlocked writes on a shared cache line, millions of mostly-duplicate entries), the background phase drained and deduped it all into a `HashSet`, and the snapshot recomputed `xxHash64(Type.FullName)` once per entity record (~5.4M redundant hashes per save on a large world) to write 9-byte tag+hash idx records plus a global `SerializedTypes.db`.
- The db's only real job was diagnostics: the string name behind "Type `<X>` was not found. Delete all of those types?" during idx loading. That map now lives in the idx itself: each `GenericEntityPersistence<T>` keeps an insertion-ordered `Type -> ushort` table, hydrated at `AddEntity` and on every deserialize path — one dictionary `TryAdd` per entity add on the game thread, amortized across gameplay, and provably immutable while the background writer reads it (adds divert to the pending queues during saves).
- idx v4 layout: the table (names only) is written before the records; records reference it by 2-byte index, shrinking from 33 to 26 bytes (−21%). The loader resolves each table name **once** (`FindTypeByHash(ComputeHash64(name))` — semantically identical to v3 resolution, `TypeAlias` included) into a constructor array, and each record becomes an array index instead of an 8-byte hash read plus dictionary probe. The unresolved-type prompt now surfaces once per type, with the name.
- Deleted outright: `World.SerializedTypes`, the drain/dedupe pass in `WriteFiles`, `BufferWriter`'s type tracking (its `Write(Type)` is now pure — payload format unchanged: tag byte + xxHash64), `FileBufferWriter`'s typeSet parameter, `Persistence.WriteSerializedTypesSnapshot`, and the adhoc db write. SerializedTypes.db is no longer produced.
- **Backward compatibility:** v0–v3 saves (including their SerializedTypes.db and legacy tdb files) load exactly as before, and every legacy load path hydrates the new table so the first v4 save after an upgrade is complete. Stale db files in existing save folders are simply ignored. Verified live: a v3 save boots, saves as v4 (Items.idx −20.2% on a dev world), and reloads with identical entity counts.
## Measured (synthetic 10M entities / 1.7GB, 64+64+32MB system blobs, 24x 2MB thick entities, dense write profile, 24 cores)
| Metric | Before | After |
|---|---|---|
| Steady-state freeze | ~740 ms | **~78 ms** |
| Main-thread publish cost | ~740 ms | **~0.1 ms** |
| Steady-state allocations | 0 | 0 (by iter 2) |
| Worker byte-load spread | 2x | ~1.15x |
The freeze is now bound by pure serialize throughput (payload / cores).
## Tests
- 779 Server.Tests + 501 UOContent.Tests pass.
- New across the branch: chunk fill/flush/owner-boundary tests, pool reuse/clear tests, an end-to-end multi-worker drain through the real wake/push/flush/pause protocol, a 50K-entry churn equivalence test for the shadow iteration re-walk (the exact pairing the snapshot writer relies on), byte-level BufferWriter output/position pins, `RuntimeLayoutIsSupported` so a silent fallback on a future runtime upgrade fails loudly in CI, and a full snapshot **round-trip test** that serializes 25K entities plus a self-payload through real workers, writes the idx/bin from the segment logs, and reloads them through the standard loader (now in v4 format).
- For idx v4 specifically: `FileBufferWriter` staging/drain/seek-patch and oversized-item tests, type-table registration tests, a hand-written v4 fixture proving an unresolvable type name skips only its own records through the console confirmation flow, and a hand-written **legacy v3 fixture** proving old saves still load and hydrate the type table for their next save.
## Trade-offs
- Chunk scheduling is nondeterministic, so worker heaps ratchet to each worker's max-ever draw rather than a fixed share. With slot ranges the balance is tight (~1.15x), so the effect is small; a shared slab pool remains an option if production shows retention creep.
- Entity-level heavy items inside slot ranges are serialized wherever they're encountered (no LPT for them); worst-case tail is one thick entity's serialize time (~ms). System self-payloads — the large ones — are still explicitly scheduled largest-first.
- Snapshot-write error granularity is per segment rather than per entity (heap-bounds bugs were the only thing the per-entity catch ever caught; idx metadata reads keep per-record granularity).
- idx v4 is a save-format version bump: old saves load unchanged through the preserved legacy paths, but saves written by this branch require this loader. Per-persistence type tables cap at 65,535 distinct entity types per boot (hard throw, orders of magnitude of headroom), and a type's table slot persists until restart even if its last entity is deleted — a few stale name entries per file, by design.
Adds an opt-in 'youngPlayerSystem.enabled' server setting (default true) that, when set to false, disables the Young player system server-wide:
- Account.Young and PlayerMobile.Young getters short-circuit to false.
- New characters no longer receive Young status or a NewPlayerTicket.
- All downstream Young checks (notoriety beneficial-action restriction, CheckYoungProtection, stealing penalties, YoungDeathTeleport, death-item movement, poison immunity, '(Young)' name suffix, CanLogout, renounce-young keyword/gump, BaseCreature.OnDeath fame penalty, OnLogin time-remaining message) become inert.
Setters are intentionally left untouched so serialized account/player flags round-trip cleanly when the setting is later re-enabled.
## Summary
- Adds proper Latin1 encoding support, replacing CP1252 usage throughout the codebase
- Adds specialized, optimized string decoding methods with safe string filtering for each encoding type
- Filters invalid Unicode characters (C0/C1 control codes, non-characters) by removal rather than replacement since
the UO client renders nothing for these characters
- Fixes UTF-16 null terminator position handling to correctly advance by 2 bytes
## Changes
TextEncoding.cs
- Added SearchValues-based invalid byte/char detection for efficient filtering
- Added encoding-specific GetString methods: GetStringAscii, GetStringLatin1, GetStringUtf8, GetStringBigUni,
GetStringLittleUni
- Each method supports a safeString parameter for filtering invalid characters
- Little-endian UTF-16 uses direct memory cast for zero-copy decoding on LE systems
- Invalid characters are removed (not replaced with U+FFFD) since the client renders nothing for them
SpanReader.cs
- Added ReadLatin1() and ReadLatin1Safe() methods
- Rewrote encoding-specific read methods to use optimized TextEncoding.GetString* methods
- Fixed UTF-16 null terminator handling: position now correctly advances by byteLength (2) instead of 1
SpanWriter.cs
- Added WriteLatin1 and WriteLatin1Null methods
## Packet Updates
- Updated all packet code to use Latin1 encoding instead of CP1252
- Affected: account packets, equipment packets, menu packets, message packets, mobile packets, player packets, secure
trade packets, vendor packets, gump packets, book packets, mahjong packets
## Filtering Behavior
Invalid characters filtered in safe mode:
```
┌───────────────┬────────────────────────┐
│ Range │ Description │
├───────────────┼────────────────────────┤
│ 0x00-0x1F │ C0 control codes │
├───────────────┼────────────────────────┤
│ 0x7F │ DEL │
├───────────────┼────────────────────────┤
│ 0x80-0x9F │ C1 control codes │
├───────────────┼────────────────────────┤
│ 0xFFFE-0xFFFF │ Unicode non-characters │
└───────────────┴────────────────────────┘
```
Note: Surrogate pairs (0xD800-0xDFFF) are not filtered because proper validation requires context checking for paired
vs unpaired surrogates. The UO client renders nothing for these anyway.
## Test Plan
- All 631 Server.Tests pass
- Verified client rendering behavior using TestUnicodeGump command (pages 1-5)
- Confirmed U+FFFD, unpaired surrogates, and non-characters all render as blank in client
- Verified Latin1 characters (0xA0-0xFF) display correctly
- Verified C1 control codes (0x80-0x9F) are filtered and don't display
### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
- Moves `CharacterCreated` event to UOContent using code generated event _CharacterCreatedEvent_.
- Moves `TargetByResourceMacro` event to UOContent using code generated event _TargetByResourceMacro_.
- Moves `GameLogin` event to UOContent using code generated event _GameLoginEvent_.
- Moves `ServerList` event to UOContent using code generated event _ServerListEvent_.
- Streamlines ProfessionInfo
> [!IMPORTANT]
> **Developer Note**
> Custom scripts that use any of these event sinks will need to be updated to use the new code generated events.
### Summary
- Removes new player "Haven Only" starting city.
- Fixed New/Old haven placement.
- Removed force-profession starting location for SA+.
- Added Royal City for SA+
- Removed Occlo for Pre-AOS.
> [!Warning]
> **Developer Warning**
> The `PacketThrottle` callback return value is now reversed. `true` indicates the connection is _throttled_.
### Summary
- Fixes an issue where connections get stalled forever
- Fixes an issue where the throttler is not working properly
- Removes account attack limiter
- Rewrites IP limiter
- Removes IP restrictions (they weren't used, and not practical)
- Fixes issue where IP limiter was counting before firewall was blocking.
View without whitespace:
https://github.com/modernuo/ModernUO/pull/1796/files?diff=split&w=1
### Summary
- Fixes various memory leaks related to spells
- Fixes Spell Plague
- Added ability to determine if `sdi` should take effect for Spell Damage.
- Fixes animal form timer ticking non-stop while logged out.
### Summary
- Fixes `[AdvancedSearch` being accessible by players 😱
- Adds `[GenCommands` to generate the same commands html page on https://muo.gg/commands.
- Fixes `[helpinfo` so all commands properly show up!
> [!WARNING]
> ### Developer Warning:
> Commands must now be registered in the `Configure` bootup phase.
> If a command is not registered early enough, it may not be available to systems like [helpinfo
> that cache their information.
> [!NOTE]
> ### Developer Note:
> Various commands related to generating content have been changed to _Developer_ and above access level.
### Screenshots
<img width="673" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/b105b5c9-5eb4-4ace-93ff-1bfb31e7132f">
<img width="547" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/e97487e8-47a5-4aa7-89cc-9fe3deda584d">
## Breaking Changes
* The Firewall and IP Limiter have been rewritten. Please read the notes carefully!
* `TcpServer.Instances` moved back to `NetState.Instances` - sorry - it was stupid to move it to begin with.
> [!Note]
> Sockets that fail the IP Limiter or Firewall will be immediately and forcibly disconnected.
> This means they will be stuck at "Verifying account..." if it was a real client.
### Summary
- Removes firewall wildcard support.
- Removes `AccessRestrictions`.
- Moves Firewall/IPLimiter to the core.
- Moves `TcpServer` to its own thread.
- Removes the `SocketConnect` and `SocketDisconnect` event sinks.
- Moves `Instances` back to `NetState.Instances`.
- Fixes a long standing bug with bad handling of duplicate listener addresses.
#### Firewall
The firewall has been completely rewritten. There is now an "Admin Firewall" which saves to the config file. Secondarily, there is an internal firewall used exclusively by the TcpServer while processing sockets. The Admin firewall mirrors it's additions/deletions to the internal firewall by adding requests to a queue.
> [!IMPORTANT]
> **Wildcard firewall entries, such as `X`, `*`, `?` are not allowed.**
> **Ranges in between IP classes or sextets are not allowed.**
> **Please make sure to use one of the following:**
> * IP Address - `192.168.1.1`
> * CIDR - `192.168.1.0/24`
> * Range - `192.168.1.1-192.168.1.100`
#### IP Limiter
The IP Limiter has been completely rewritten. The available configurations are:
```json
"ipLimiter.enable": "True",
"ipLimiter.maxConnectionsPerIP": 10,
"ipLimiter.clearConnectionAttemptsDuration": "00:00:00:10",
"ipLimiter.clearThrottledDuration": "00:00:02:00",
```
The IP Limiter is set up to prevent spamming connections from the same IP. Every time an IP connects, it is added to a connection list. After 10 attempts, the IP is added to the throttle list. To keep the system fast, the connection list is entirely wiped every 10 seconds, and the throttle list is entirely wiped every 2 minutes.
### Summary
- Moves starting city info from AccountHandler to CharacterCreation
- Simplifies the logic of determining the starting cities
- Adds support for Trammel & Felucca for non-young accounts
- Fixes fall through starting city for v6+ in UOR era.
- All staff are force-sent to GA.
Closes#1408
### Summary
.NET 8 supports Xoroshiro 256** off the shelf and added Shuffle. Switching to that implementation.
### Developer Notes
* Removed many convenience methods that weren't used.
### Summary
- Fixes usernames not being `Intern`ed
- Reverts methods related to getting accounts from returning `Account` to `IAccount`.
- Makes `IAccount` also `ISerializable`
- Adds `IGenericReader.ReadAccount()` and `IGenericWriter.Write(IAccount)` -> The read method supports the original serialization of username, and using `IAccount.Serial`. The write method only serializes the `Serial`.
- Exposes `ReadStringRaw()` to allow some advanced scenarios.
### Summary
- [X] Fixed a bug where entity persistence was serialized out of order, causing world corruption.
- [X] Fixed LastSerialized not being utilized properly and dangling references still becoming an issue.
- [X] Added a new `GenericEntityPersistence<T>` type to encapsulate `ISerializable` serialization.
- [X] Removing the custom logic and moved Items, Mobiles, Guilds, and Accounts to GenericEntityPersistence.
- [X] Changed serialization to use the singleton pattern to reduce calling methods from stored variables.
### Summary
Adds a generic entity persistence. This can be used to create new entity types that have a `Serial`.
Here is an example:
```cs
public class BOBEntries : GenericEntitySerialization<IBOBEntry>
{
public static void Configure()
{
Configure("BOBEntries");
}
}
```
The annotation tells the system what folder to serialize the entries to. The class/interface (`IBOBEntry`) is the root type that implements `ISerializable`.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
## MAJOR CHANGE (API BREAKING)
Added a player murder system to facilitate reporting murders. This should make it easier to extend to create a bounty system or other related game content. Player murders will be saved in a folder called _PlayerMurders_.
### Motivation
The motivation was two-fold, performance, and bug fixes.
First, murders are one of two systems that do a pre-world-save check on _every mobile in the game_ to decay kills and set their expiring murders. This is taxing since it freezes the world and makes world saves take longer. Every mobile has ShortTermMurders even though it is a player concept. And next, 90%+ of players are not murderers but had an ever increasing MurderElapse time that was being tracked against GameTime. These properties were also serialized unnecessarily for all mobs.
Second, when I tried to optimize/refactor the code, it was obvious that the system has bugs.
### Major API Changes
- [X] Created a player murder system and moved `ShortTermMurders`, `ShortTermElapse`, and `LongTermElapse` to the system.
- [X] Added convenience property `PlayerMobile.ShortTermMurders`.
- [X] Added convenience properties `PlayerMobile.ShortTermMurderExpiration` and `PlayerMobile.LongTermMurderExpiration`
- [X] Moved ReportMurdererGump.cs
- [X] Adds an `EventSink.PlayerDeleted` event.
### Notes
The system currently does not support NPCs. To support expiring murders on NPCs I highly recommend a different architecture for large servers (500k+ mobs including players). Specifically switching from looping through all MurderContext to a time-order link list.
* Movie ignore mobiles to Mobile class
* Allow necromancer familiars to ignore mobiles
* ChampionSpawn should not quietly fail when creating new spawn
* Fix client party crash bug: 2 people partied, the leader logs out, other client is hung
* Fix spellbooks creating with magery/meditation only and creating magery multiple times
* Fix BaseRunicTool.GetRandomSlayer() creating more undead slayers than intended
* Do not allow players to dismount each other whilst mounted
* Fix AOS onwards damage increase tooltip and wrong formula in AOS
* Fix monsters killing other monster revenants + animate dead should not attack player pets + other animates
* Fix fire steed having loot pack added twice
* Fix oaks spawn not able to create Unicorns + Kirins via Activator.CreateInstace() due to constructor having name as parametr
* Fix ignoreMobiles flag not propagated to client for non-players.
* Fix harrower tents not leeching from players
* Fix boats speedhacking around the map
* Fix bless, agility etc. not renewing duration
* Fix reveal always worked
* Fix empty constructor for steeds so they don't throw now.
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
* Adds `SerializationProperty` - This will generate the private variable for serialization. Also provides an option `useField` to specify your own. Example:
```cs
[SerializationProperty(0, useField: nameof(_resource)]
```
- [X] Fixes NPE from account tags.
- [X] Fixes bad skill check due to missing cast to double.
- [X] Fixes water elemental duration.
- [X] Standardizes spell summon duration by expansion.
