Follow-ups to #2559, from a review of the ported idle-sleep/scheduler-health changes.
### Fixes
- **`NetState.IsIdle` omitted `_pendingDisconnects`** — `Slice()` drains five queues; the property checked four. The other deferred work (`_connectingQueue`, alive checks, movement throttle) is time-gated and correctly excluded; the disconnect queue was the only ready-work omission. Impact was bounded (≤ one idle wait of delay), but the property's contract is "sleeping cannot strand pending work".
- **Neither new setting was clamped** (`Main.cs`):
- `server.lateWakeThreshold: -1` made `late <= threshold` false for every sample even at zero late wakes, so from the second sample on, sleeping was re-suspended every second, forever — a permanent full-core spin whose only trace was a nonsense warning ("… at least 8ms late 0 time(s)").
- `server.eventLoopIdleWaitMs: -1` disabled sleeping while the admin gump reported **Healthy** (it tested `== 0`).
- Both now clamp to `>= 0` and log a warning naming the configured value. `-1` is a natural thing to reach for given the sibling key's doc says "set very high to disable".
- **World snapshots were misattributed to `StolenMs`** — `World.Snapshot` ran outside all five profiler phases, so a 3-second save inside a sample read as ~75% stolen, and `debugging-event-loop.md` teaches stolen = "the host ran something else". The diagnostic pointed operators at buying dedicated CPU for their own largest loop-thread stall. Saves now land in a new `WorldSnapshot` phase; `[LoopStats` iterates `PhaseCount` generically, so the report and CSV pick it up with no changes.
- **Admin gump conflated host-forced spin with configured spin** — when the startup probe finds no high-resolution wait support it zeroes the idle wait, after which the gump said "Spinning (configured)" and the operator's config said 2. New `Core.IdleSleepUnsupported` property; the gump now shows "Spinning - host cannot honor short waits" as a distinct fourth verdict. A genuinely configured 0 still reads "configured" (the probe only runs when the configured value was > 0).
- **The backoff-ceiling `Error` logged once per process lifetime** — `_loggedBackoffCeiling` never reset, and at the ceiling the method returns before the `Warning`, so a host that recovered (>60s clean streak) and later degraded back to the ceiling never re-logged the one operator-actionable message. The flag now resets with the clean-streak escalation reset.
- **Removed the unreachable "already suspended, extend" branch** — no sleeps occur while suspended, so `_lateWakes` stays 0 and every suspended sample early-returns before reaching it; with the threshold clamped it can never fire. If sleep gating ever changes, the normal path handles the case by counting a fresh episode.
`dev-docs/debugging-event-loop.md` updated to match (phase list + gump verdict table).
### Verification
- `dotnet build` clean (0 warnings) both normally and with `-p:EventLoopProfiling=true` (the snapshot phase only becomes live IL under the profiling flag).
## Problem
`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.
## Result
Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:
| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |
**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.
## How
The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.
**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.
**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.
## Configuration
| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |
## Diagnostics (compiled out by default)
`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.
## Verification
- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.
## Notes
The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
## Why
The shard owner, on a Starlink CGNAT address, was blocked by the imported reputation blocklist.
The cause was not CrowdSec. The address was a literal line in `ip-blocklist.txt`, so `BlocklistFilter` denied it at accept and then promoted it — and clearing the CrowdSec decision could not fix it either, because the file entry re-reports within `promoteSuppression` of every reconnect attempt.
This is structural, not a one-off. Reputation feeds list shared consumer address space constantly: on CGNAT one public address fronts many subscribers **at the same time**, so a single abusive customer gets the address listed and everyone else behind it is blocked with them. Where leases rotate, a listing says little about whoever holds the address now. Around 1,000 Starlink addresses sit in the current list.
So exemptions go where they cost nothing, and escalation is driven by what a connection actually does.
## Generator — `tools/Export-IpBlocklist.ps1`
`-AllowlistFile` takes multiple paths, subtracted from the merged set before the output is written. Defaults to every `ip-allowlist*.txt` beside the output, merged into one allow set:
- `ip-allowlist.txt` — operator exemptions, created once and **never rewritten**
- `ip-allowlist-<name>.txt` — a carve-out you built, regenerable and copyable between shards
**Subtraction is range-correct.** An allowlisted address inside a blocked CIDR splits that CIDR around the hole rather than being silently ignored. This also fixes `-ExcludeAnonymizers`, which parsed CIDR entries into `$anonCidr` and then only ever subtracted singles.
**No carve-out ships.** A carve-out names a real network, and which ones a shard should exempt depends on where its players actually are — so publishing one would make that policy call for every shard and put a specific provider's address space in the repo. The script builds them on request instead:
```powershell
.\Export-IpBlocklist.ps1 -AddCarveout starlink -Asn 14593
```
Carve-outs are **discovered, not configured**: every `ip-allowlist*.txt` beside the output is subtracted, by the generator and by the shard, so a file an admin adds needs no config edit and no code change. Each carries an `asn=` marker in its header, which is how `-RefreshCarveouts` rebuilds it without the script keeping a list of anyone's networks; a hand-written allowlist has no marker and is never rewritten.
Prefixes come from **announcements, not ownership records**, because registry data disagrees with what is actually routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and `206.83.96.0/19` reads as APNIC in RDAP even though `206.83.96/21` is announced by Starlink.
Editing an allowlist bypasses `-MinInterval`, so a just-added exemption isn't indistinguishable from the allowlist not working. A Starlink carve-out, if you build one, costs **~4,300 IPs + ~144 CIDRs of 4.2M (0.10%)**.
## Allowlists
**`FileAllowlist`** reads the same files the generator subtracts, so an operator entry means "leave this address alone" for real. Subtraction alone only covers being *blocked*; behavioural detections never consult the blocklist, so without this a carve-out was quietly routed around — one scanner behind a shared address was enough to get everyone behind it contributed and firewalled, with nothing in the shard's own config explaining why. Reading the files also means an entry applies on the next reload rather than the next regeneration, which is what matters when someone is complaining now.
**`LoginAllowlist`** is earned by authenticating, with a 90-day TTL because an address that logged in years ago is a stranger. Its own store rather than `Account.LoginIPs`, which has no timestamps and cannot be backfilled. An entry is evidence rather than a licence: 10 suppressed contributions in an hour revokes it, and a fresh login forgives the tally.
Both are consulted **only after the blocklist has already matched**, so a normal accept pays nothing for them and the accept gate stays allowlist-free. `BanExemptions` combines them behind `BanChannel.IsExempt` and suppresses escalation only — every local defence still applies.
Two limits, both deliberate and documented in the class: `LoginAllowlist` **cannot bootstrap** (an entry is only earned by getting in, so it never repairs an existing false positive), and it is weakest on rotating CGNAT. That is why `FileAllowlist` is the fix for those, and why it is manual.
## Behavioural detection
| Reason | Trigger |
|---|---|
| `silent-connect` | Reaped after 5s having sent **zero bytes** |
| `invalid-seed` | Opened with a zero seed |
| `foreign-protocol` | Positively identified as HTTP, TLS or SSH |
**`ForeignProtocol` inverts the test.** Asking "is this a good UO client?" cannot work: `LoginEncryption.ClientDecrypt` is a byte-for-byte stream XOR, so a legitimate client with encryption enabled when the shard expects none sends a structurally perfect connection whose payload is noise. "Speaks HTTP" is safe where "unreadable" is not — however misconfigured a UO client is, it never sends `GET / HTTP/1.1`.
Nothing assumes arrival framing. TCP has no message boundaries, so a rule of the form "these bytes must arrive together" is broken by construction and drops real players on poor links. A prefix match with too few bytes to confirm waits for more. A four-byte seed can legitimately spell `GET ` (the address 71.69.84.32) or `0x16 0x03 0x0?` (22.3.x.x), so confirmation requires the request line to continue in printable ASCII or an actual ClientHello inside a plausible record — a real client's fifth byte is a packet id (`0x80`, `0x91`, `0xEF`), none of them printable, so those collisions fall through.
Everything is keyed on **bytes-received rather than elapsed time**. A connection that sent something and ran out of time is far more likely a slow link than an attack, and banning those produces the worst failure mode available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being firewalled off.
## `AutoDenylist`
A short-lived local hold (15m) on behavioural detections, as `IConnectionFilter` + `IBanReporter` over one store so the engine detection sites never reach into content.
This closes the gap where a flood pays for a socket, buffer and `NetState` slot per connection while waiting for the OS bouncer — the verdicts that matter most are reachable only *after* reading bytes — and it is the entire defence on a shard running no bouncer, which is the default config. Not persisted: a holding pen that survives restarts is a ban without a ban's review.
Cost: one dictionary lookup on a usually-empty dict per accept.
## `BanReasons`
Centralises the reason slugs. `IsBehavioral` is an **opt-in** set, not "everything except manual", so a future reason escalates normally instead of silently inheriting an exemption or entering a local denylist.
This caught a real bug during review: the first cut of the exemption swallowed `manual` admin bans (`Commands.cs`, three sites in `AdminGump`) for any allowlisted address.
## Fixes found in review
- **`BanConfiguration.Settings` was null until `Configure()` ran**, while the reap path dereferences it every `Slice()`. A harness driving `NetState.Slice()` directly hit an NRE that presented as flaky because it depended on whether an earlier test had already called `Configure()` — which is why it failed on some CI platforms and not others. Now starts at the record's defaults, with idempotency tracked by a flag; this also removes the same latent NRE from the pre-existing rate-limit path.
- **`-AllowlistFile` was typed `[string]`** while documented and used as a list, so passing two paths would have collapsed them into one string.
## Layout and docs
Content network code moves out of `Misc/` into `UOContent/Network/`, one concern per folder — `AutoDenylist/`, `Blocklist/`, `CrowdSec/`, `Firewall/`, `LoginAllowlist/`, `Packets/`. **Namespaces are untouched**, so these are pure file moves (git tracks all 16 as renames).
`dev-docs/ip-bans-and-allowlists.md` documents the subsystem, leading with the operator process for unblocking a player — including the three things that look sufficient and are not: deleting the CrowdSec decision alone, editing `ip-blocklist.txt` by hand, and `cscli allowlists` alone. `.gitignore` covers the new config files.
## Testing
Build clean. **Server.Tests 810 passed**, **UOContent.Tests 637 passed**, zero warnings. This branch adds 38 tests; the rest of the delta is main's, since this is rebased on current `main`.
New coverage: TTL boundary and renewal, private-address exclusion, manual-ban-never-exempt, unopted-reason-never-exempt, strike revocation, quiet-window reset, login forgiveness, file-allowlist CIDR coverage, file-allowlist not spending the earned list's strikes, denylist expiry-on-read, cap enforcement, lapsed-entry reclaim, HTTP/TLS/SSH identification, seed-collision fall-through, and encrypted-login-is-not-foreign.
Generator verified end-to-end against live feeds: a clean run ships no carve-out, `-AddCarveout starlink -Asn 14593` fetches and collapses 213 prefixes to 115 ranges in 0.1s over 4.2M entries, `-RefreshCarveouts` rediscovers it by its `asn=` marker, a hand-written allowlist is left untouched, and deleting a carve-out drops it rather than having it rewritten. CIDR splitting verified exhaustively: a single-IP hole in a /24 leaves exactly 255 of 256 addresses blocked.
## Operator note
Existing installs are unaffected until the generator next runs, which creates `ip-allowlist.txt` and nothing else. To unblock someone: add the address to that file and delete any live CrowdSec decision — the existing ban outlives the config change. The shard picks the entry up on its next reload, so re-running the generator is optional.
A shard whose players are on CGNAT (satellite, mobile, or an ISP short on IPv4) will likely also want `-AddCarveout`; see `dev-docs/ip-bans-and-allowlists.md`.
## Also included: a latent CI failure this PR surfaced
`fix(tests): serialize test classes that rent through STArrayPool` touches a property-list test file that has nothing to do with this feature. It is here because it was failing macOS CI, and it is trivially cherry-pickable out if you would rather it went to `main` on its own — **which may be the better call, since it is failing `main` today.**
CI has since gone green with it applied.
`STArrayPool` is single-threaded by design and its bucket cache is a plain `static`, not `[ThreadStatic]`, with a check-then-act initialize in `Return()`:
```csharp
var cacheBuckets = _cacheBuckets ?? InitializeBuckets();
```
Two threads both see null, both initialize, and the loser trips `Debug.Assert(_cacheBuckets is null)`. Anything renting from it has to stay off parallel test threads — which is what the `DisableParallelization` collections are for.
- `ObjectPropertyListReentrancyTests` and `ObjectPropertyListNestedBuildTests` (added in #2555) build property lists, which rent the interpolation buffer, but were not in the sequential collection — unlike `PropertyListInvalidationDuringBuildTests` in the same file. This is a **latent failure already on `main`**; it is timing-dependent, so it shows on some platforms and not others.
- `AutoDenylistTests` (added here) has the same exposure: its cap tests reach `AutoDenylist.Sweep`, which rents a `PooledRefList` without `mt`. The blocklist tests need no marking because `BlocklistSnapshot.Build` asks for the `mt` pool explicitly.
No production change — `STArrayPool` is the right pool on the game loop, where both `Sweep` and the property list actually run.
## Deliberately not included
Waiting for a fragmented four-byte seed at `AwaitingSeed`. It looked like a bug but the disconnect is a deliberate defence: only pre-0xEF clients reach it (0xEF goes through `HandlePacket`, which already waits for its 21 bytes), and waiting converts an instant drop into a full 5s slot hold for a client sending one or two bytes, or a loris dribbling a byte every few seconds. Against a fixed 4096-entry `MaxConnections` table that trades capacity that matters for a fragmentation case a reconnect already fixes.
## Summary
Two related fixes on the outbound path:
1. Consume **IORingGroup 1.0.8**, which allows more than one send in flight per socket, and expose the two settings that go with it.
2. Stop `NetState.Send` silently discarding packets when the send buffer fills — including an out-of-bounds write reachable in that state.
## 1. Send-path stall (RIO)
RIO reports send completion on **acknowledgement**, not on copy, so a completion cannot arrive sooner than one round trip. With one send in flight, `PostSend` refused to post again until the previous completion arrived — capping a connection at **one send per RTT** whenever it had data queued.
Measured on a 50ms-RTT production shard:
| | before | after |
|---|---|---|
| in-game latency, data flowing | **101–146 ms** | **48–51 ms** |
| p95 | ~135 ms | 52.8 ms |
| samples > 70 ms | 20 | **0** |
The control that confirms the mechanism: server-side post→completion was **unchanged** at median 92ms across both runs. The ACK-binding is inherent to RIO and did not move; only its propagation into application latency did.
Two things worth recording, because they explain why this went unnoticed:
- As little as **6 bytes** of queued data held the gate shut, so it reproduced in empty areas, not just crowded ones.
- The same measurement at loopback RTT is **microseconds**, so local testing could never surface it.
New settings, both restart-time:
- **`network.maxOutstandingSends`** (default 32) — sends in flight per connection. Honoured by RIO only; other backends complete sends on copy and report 1. Costs a request-queue and completion-queue slot per send, **not another buffer**, since every outstanding send addresses a different range of the same registered buffer. Worst-case added latency is roughly `completion RTT / value`.
- **`network.sendBufferSize`** (default 256KB) — per-connection send buffer, coerced to a power of two of at least the platform allocation granularity. This is the lever for the disconnects below, and the per-connection memory ceiling.
## 2. Send buffer full
`NetState.Send` had three failure modes once the buffer filled, none of them visible:
| writable | behaviour |
|---|---|
| `0` | `GetSendBuffer` returned false → **packet dropped**, no log, no disconnect |
| `4 … needed-1` | `Compress` returned 0 → `CommitWrite(0)` → **packet dropped** the same way |
| `1 … 3` | `safeOutputLength = (nuint)output.Length - 4` **underflows** → hot-loop bounds check never trips → **writes past the span** |
The first two leave a client connected while quietly missing game state, which is undiagnosable from either end. The third corrupts the in-flight region of the ring buffer, and is reachable precisely when a connection is congested, since callers only check for non-zero space.
`Compress` now refuses an output too small to bound, and `Send` reports exhaustion instead of dropping — logging and disconnecting with **needed / writable / unacked / capacity**. Those numbers separate a slow client holding the buffer from a buffer genuinely too small for the shard, which is the case that warrants raising `network.sendBufferSize`.
## Testing
`NetworkCompressionBoundsTests` covers the underflow using sentinel bytes around the output window. **Verified to fail without the guard** (4 failures from overwritten sentinels), confirming the out-of-bounds writes were real rather than theoretical.
Full suites green: **788 Server.Tests**, **597 UOContent.Tests**, Release build clean against the published 1.0.8.
## Notes for reviewers
- Upstream change: modernuo/IORingGroup#9.
- The buffer-full path is now *loud* where it used to be silent. If a shard has been quietly dropping packets under load, this will surface as disconnects — that is the intended outcome, and the log line says which setting to raise.
- Follow-up under discussion: promoting a connection to a larger buffer instead of disconnecting, which looks feasible on a live connection since buffers are referenced per-operation rather than bound to the request queue.
Reshapes IP banning around one idea: **core owns the question, content owns every answer.**
Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter.
## The seam
```csharp
public interface IConnectionFilter
{
string Name { get; }
void Configure();
void Start(CancellationToken token);
void Stop();
bool ShouldDeny(IPAddress address);
}
```
The accept path went from hardcoded branches to one question:
```csharp
else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy))
{
logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy);
}
```
Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS.
Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's.
A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either.
This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic.
## What ships behind it
**`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands.
**`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall.
The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up.
The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves.
**`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream.
## CrowdSec: contribute-first
`IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`.
Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work.
### Why not pull decisions from CrowdSec?
The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce.
The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline.
## Threading policy
`CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem:
- Anything touching game state runs **only** on the main loop.
- Heavy work that *needs* game state must be **chunked** across ticks, never threaded.
- Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**.
Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it.
## Shared primitives
`SortedRangeIndex<T> where T : IBinaryInteger<T>` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it.
Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests.
`IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated.
## Config
| File | Owner | Keys |
|---|---|---|
| `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` |
| `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` |
| `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` |
| `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) |
Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change.
## Notes for review
- **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex<UInt128>`, same semantics and public API.
- **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`.
- **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core.
- **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel.
- `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR.
- The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message.
## Tests
**1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader.
## Summary
- **Timer-aware idle sleep**: Exposes `Timer.MillisecondsUntilNextTick()` to calculate remaining ms until the next timer wheel tick (0–8ms). The game loop sleeps for that duration minus a 1ms safety margin, instead of spinning at 100% CPU.
- **I/O completion wakeup**: Replaces `Thread.Sleep` with `NetState.WaitForCompletion()`, which uses platform-native completion notification (RIO `RIONotify` on Windows, `eventfd` on Linux, `kevent` timeout on macOS) to wake immediately when network data arrives during sleep.
- **Always-on**: Removes the debug-only `core.enableIdleCPU` config gate. The sleep is self-regulating — under load, `MillisecondsUntilNextTick` returns 0 so no sleep occurs (zero overhead). On idle, CPU drops from ~100% to ~1%.
- **CPS calculation cleanup**: Replaces the 128-element ring buffer with an EMA (exponential moving average) for `CyclesPerSecond`/`AverageCPS` — fewer allocations, no LINQ `.Average()` call each sample.
- **IORingGroup 1.0.6**: Adds `WaitForCompletion(int timeoutMs)` to the `IIORingGroup` interface with platform implementations:
- **Windows**: `RIONotify` arms the CQ event, `WaitForSingleObject` with timeout
- **Linux**: `eventfd` registered with io_uring, `poll()` with timeout
- **macOS**: `kevent()` with timeout
## Test plan
- [ ] Build succeeds on all platforms (`dotnet build`)
- [ ] Empty server: verify CPU usage drops from ~100% to ~1% idle
- [ ] Loaded server: verify no added latency — `MillisecondsUntilNextTick` returns 0 when timers are firing, sleep is skipped
- [ ] Connect a client during idle — verify connection accepted within one timer tick (~8ms)
- [ ] Verify `[admin` gump shows reasonable CPS values (EMA convergence)
## Summary
Server-side movement throttle that prevents speed hacking while accurately identifying cheaters with detection of lagging connections.
**Key features:**
- Credit buffer (200ms) absorbs timing jitter from legitimate players
- Movement queue handles larger bursts, draining at proper game-tick intervals
- RTT measurement distinguishes network lag from speed hacks
- Queue depth detection catches ACK-throttled speed hacks (going straight)
## How It Works
**Throttle** (prevention): Movements arriving too early either consume credit or get queued. The queue drains at
correct intervals, so speed hackers can't move faster regardless of what they send.
**Detection** (identification): Combines multiple signals to identify cheaters:
| Signal | What it catches |
|--------|-----------------|
| Queue depth ≥4 sustained | ACK-throttled speed hacks (client limits unacked moves to 5) |
| Movement rate >1.05x | Direction-change speed hacks where timing is visible |
| Stable RTT + high queue | Eliminates false positives from laggy players |
**RTT-Aware Logic:**
- Probes only sent to players actively moving (event-driven, not global loop)
- Stable low-latency + problems = suspicious
- Unstable/high-latency + problems = probably just lag, throttle handles it
## Configuration
```json
{
"movementThrottle.maxCredit": 200,
"movementThrottle.softQueueLimit": 6,
"movementThrottle.hardQueueLimit": 10,
"movementThrottle.debugLogging": false
}
```
### Summary
- Bump IORingGroup 1.0.0 → 1.0.1 — fixes a disconnect handling bug in the native ring layer
- Fix ghost NetStates — Dispose() set _running = false before checking it, so the "force immediate disconnect" path
was dead code. Capture wasRunning before clearing it, add [Obsolete] guard, and route internal callers through
DisposeInternal()
- Fix unauthenticated socket cleanup — graceful disconnect on unauthed connections could get stuck with pending sends;
now force-immediate after Disconnect() if DisconnectPending is already set
- Replace ConcurrentQueue<NetState> _disposed with Queue<NetState> — server is single-threaded; moved the field into
the Network partial class where it's consumed
- Move ConnectingSocketIdleLimit into the Network partial class alongside DisconnectUnattachedSockets
- Reset activity timer on receive, not just send — receiving data directly proves liveness instead of relying on the
ping→pong→send round-trip to reset the timer
- Move CheckAllAlive from Timer into Slice — the timer fired before I/O completions were processed, so after server
stalls (world saves), buffered client pings hadn't reset timestamps yet, causing false disconnects. Now runs at the
end of Slice() after all recv completions are handled
- Lower inactivity timeout 90s → 30s, check interval 90s → 5s — clients ping every ~1s, so 30s of silence is ~30
missed pings; worst-case detection drops from ~180s to ~35s
- Simplify CheckAlive — early-return when socket is null or alive; force-kill stuck DisconnectPending sockets
immediately instead of calling Disconnect() again
- Remove unused imports from GameEncryption.cs
> [!IMPORTANT]
> **Breaking Changes**
> - DecodePacket and EncodePacket delegates replaced with IClientEncryption interface
> - NetState.Connection (Socket) replaced with internal RingSocket management
> - NetState.RecvPipe and NetState.SendPipe removed (buffers managed internally)
## Summary
Upgrades the networking stack from PollGroup-based I/O to io_uring, significantly improving I/O performance on Linux.
This also adds native client encryption support for encrypted UO clients.
## Major Changes
io_uring Networking Architecture
- Replaced PollGroup with IORingGroup for async socket I/O operations
- Removed Pipe.cs (mirrored ring buffer) and TcpServer.cs in favor of RingSocketManager
- Added NetState.Network.cs - centralized network infrastructure handling accept, recv, send, and disconnect
completions
- Added SocketHelper.cs - platform-specific socket utilities for raw socket handle operations (getpeername,
getsockname)
- Buffer management now handled by RingSocketManager with configurable slab allocation
### Client Encryption Support
- Added full encryption stack in Network/Encryption/:
- EncryptionConfig.cs - configurable encryption modes (None, Unencrypted, Encrypted, Both)
- EncryptionManager.cs - encryption detection and initialization for login/game packets
- LoginEncryption.cs - handles login packet encryption with version-derived keys
- GameEncryption.cs - handles game server encryption using Twofish
- TwofishEngine.cs - optimized Twofish block cipher implementation
- LoginKeys.cs - encryption key table for client versions
- IClientEncryption.cs - interface for client encryption implementations
### NetState Improvements
- Replaced Socket Connection with RingSocket _socket for managed socket lifecycle
- Changed from GCHandle polling to event-based completion processing
- Disconnect handling now properly waits for pending sends to flush
- Simplified connecting socket management using lazy queue removal
### Configuration
- New settings: network.encryptionMode and network.encryptionDebug
- Encryption mode flags: Unencrypted, Encrypted, or Both
### Dependencies
- Replaced PollGroup NuGet package with IORingGroup
- Linux requires liburing-dev / liburing-devel package
### Test plan
- Verify server starts and accepts connections on Linux with io_uring
- Verify server starts and accepts connections on Windows (fallback to IOCP)
- Test unencrypted client connections (ClassicUO with encryption disabled)
- Test encrypted client connections if available
- Verify graceful disconnect flushes pending data
- Confirm CI builds pass on all target platforms
Kamron Batman