2dbaa87377
Two features ran on every shard out of the box, each polling on its own 60s timer for files most shards never generate, neither ever asked for. Fixing that turned into untangling why they shared a config file — and then into the on-loop cost of the three lists behind them. ## Before / after Measured on the shipped defaults. On-loop numbers are what freezes the world; the tick budget is 8 ms. | | before | after | |---|---:|---:| | Blocklist poll on a shard with no list | every 60s, forever | **none** (opt-in) | | Manual allowlist poll on a shard with no carve-outs | every 60s, forever | **none** (opt-in) | | Promote-guard sweep timer | leaked on `Stop()` | stopped, and only started when hits are reported | | Login allowlist flush, on-loop | O(n) walk + 2 arrays **every 60s**, LOH past ~5,300 entries | reused buffers, **hourly**, zero steady-state allocation | | Auto-denylist, accept path | 9.1 ns/call | **6.1 ns/call** | | Auto-denylist, sustained flood at cap (60k rejected) | 26.7 ms | **9.3 ms** | | Auto-denylist, flood end — **worst single call** | 9.49 ms | **0.05 ms** | | Auto-denylist cap | 65,536 (stranding 9,895 slots) | **324,449** (exact `HashSet` capacity, ~19 MB) | The auto-denylist row that matters is the third: the on-loop stall at flood end drops **190×**, because retiring lapsed holds is now the number expiring rather than the number held. ## Why this design It is built for the shape of attack these shards actually see: **hundreds to a few thousand connections per second**, occasionally tens of thousands, sustained over minutes rather than delivered instantly. Against that shape the cap now covers the whole observed range (50k–250k distinct sources) in memory, and the work of expiring them spreads across the accept calls that were already happening. There is one case this design is *worse* at than the old one: if every held entry lapses within the same millisecond, retiring them costs ~10.7 ms against the old ~8.9 ms, because the ring's random-access set removals lose to a sequential dictionary scan. Reaching it requires an entire flood to arrive inside one millisecond. **A shard absorbing 324,449 connections in a millisecond is finished at the accept path no matter what this list does** — that is the point where the answer is upstream security and scrubbing (an L4 proxy, edge filtering, a bouncer at the kernel), not a data structure in the game loop. We chose the design that fits the attacks we see and degrades honestly past them, rather than over-engineering for one we do not. ## Blocklist — now opt-in `BlocklistFilter.Start` only bailed when `_path == null`, which needs `file` to be empty. The default is `"Configuration/ip-blocklist.txt"`, so on any default install both `Task.Run(PollLoop)` and a recurring `SweepGuard` timer started unconditionally, logging *"Blocklist inert: no list at …; polling every 60s"* and then doing exactly that forever. Adds `"enabled"`, default `false`, using the `_enabled = s.Enabled && <preconditions>` idiom already in `LoginAllowlist` and `AutoDenylist`. **Upgrade is deliberately loud**: a missing key binds to the default, so `LogWhyDisabled()` splits three cases and a shard with a list on disk but no `enabled` key gets a **Warning**, not silence. ## `FileAllowlist` → `ManualAllowlist`, with its own config Moves to `Configuration/ip-allowlist.json` (`enabled` default `false`, `files`, `reloadInterval`) and into `Network/ManualAllowlist/`, mirroring `Network/LoginAllowlist/`. It was never a sub-feature of the blocklist. `ManualAllowlist.Contains` has two callers: | Caller | Could anything else do it? | |---|---| | `BlocklistFilter.Evaluate` | **Yes** — the generator already subtracts these files at generation time | | `BanExemptions.IsExempt` | **No** — sole mechanism for suppressing behavioural ban contributions | The second reaches `BanChannel.IsExempt` with no blocklist in the path. A shard running **no blocklist** still needs this so the admin's own IP isn't auto-banned by rate-limit detection, so a shared flag couldn't express it — the implication is asymmetric. They still work together via a startup warning when the blocklist is on and the allowlist is not. On the name: "File" described the storage. The distinction from `LoginAllowlist` is **provenance** — declared by an operator versus earned by authenticating — and "Manual" matches `BanReasons.Manual`. `allowlistFiles` is removed from `BlocklistSettings` outright; blocklists have not shipped long enough for anyone to have set it. ## Login allowlist flush `Flush()` allocated two arrays sized to the live entry count and copied the whole dictionary into them **on the game loop**, every 60s. `UInt128` is 16 bytes, so past ~5,300 entries that first array was an LOH allocation once a minute, forever. The file write was already off-loop; the walk was not. Static buffers grown geometrically; the writer owns them until it posts completion back through `Core.LoopContext`, so `_writing`/`_dirty` stay loop state (rule #10). Interval → 1 hour against a 90-day TTL. Clean shutdown writes synchronously via `EventSink.Shutdown`; `HandleClosed` skips `InvokeShutdown` when crashed, so the crash path subscribes separately and only writes when it is actually on the loop thread. Also fixes a pre-existing hole where `_dirty` was cleared *before* the write, so a failed write dropped entries despite the comment promising a retry. ## Auto-denylist: expiry ring Reclaiming lapsed holds was O(entries held) — every cap-triggered reclaim during a flood walked the whole dictionary to find the few that expired, and `_warnedFull` suppressed the log, not the work. A hold is **never refreshed** now: the first detection sets the expiry, later ones leave it. That makes insertion order equal to expiry order, so a ring of the same keys is sorted by construction and retiring stops at the first live record. Nothing is lost — the rate limiter runs *ahead* of the connection filters (`NetState.Network.cs`) and reports to the ban channel, so a flooder whose hold lapses is re-held on its next attempt. Because the ring carries the expiry, the membership side only answers "present?", so it is a `HashSet` — measured at **36 B/slot against the dictionary's 52**. `HashSet` and `Dictionary` share `HashHelpers`, so the from-empty capacity progression is identical (36,353 → 75,431 → 156,437 → 324,449 → 672,827) and the cap still lands on one exactly. The ring is parallel `UInt128[]`/`long[]` rather than an array of structs — `UInt128` forces 16-byte alignment, so a packed pair costs 32 bytes where these cost 24, and the drain reads only the `long[]`. Rejected after measuring: splitting the drain into a scan loop plus a removal loop (inside noise — both issue N hash removes, and the pointer math was never the bottleneck), and `Dictionary<UInt128,bool>` with tombstoning instead of removal (10% slower *and* unbounded, which breaks the cap). ## Testing Build clean, 0 warnings. **1,530 tests pass** — 708 UOContent, 822 Server. Tests were reworked rather than patched: the refresh test inverts to `Repeat_detection_does_not_extend_the_hold`, the obsolete sweep-throttle test is deleted along with the throttle, and four were added for the ring — set/ring parity, release-then-re-hold not being retired by the stale record, exact fill of a non-power-of-two cap, and the moved allowlist config's casing contract. The throttle test added mid-PR was verified to fail without its fix before being deleted. One commit is comments only (verified: a diff filtered of `//` lines is empty), removing development narration — a `"(Task 2)"` plan reference, `"matching the per-feature JSON config pattern used by X"` across four loaders, a duplicated threading note — and repointing `Firewall` at `dev-docs/ip-bans-and-allowlists.md` instead of a "ban-channel design doc" that does not exist. Note `Distribution/Configuration/blocklist.json` is gitignored (`.gitignore:14`) and generated from the record defaults on first boot, so the record default *is* the shipped default.
236 lines
13 KiB
Markdown
236 lines
13 KiB
Markdown
# IP Bans, Blocklists and Allowlists
|
||
|
||
How a shard decides to refuse a connection, how it contributes bans to an external bouncer, and how an
|
||
operator exempts someone who was caught by mistake.
|
||
|
||
If you are here because **a player cannot connect**, skip to [Unblocking a player](#unblocking-a-player).
|
||
|
||
## The shape of it
|
||
|
||
Two independent questions, deliberately separated:
|
||
|
||
| Question | Answered by | Effect |
|
||
|---|---|---|
|
||
| Refuse this connection? | `IConnectionFilter` implementations, at accept | The socket is dropped |
|
||
| Tell the outside world about it? | `BanChannel` → `IBanReporter` implementations | CrowdSec, and from there an OS bouncer |
|
||
|
||
`BanChannel` **never enforces** and filters **never report on each other's behalf**. Enforcement that
|
||
outlives the process belongs to the OS bouncer; the shard only contributes.
|
||
|
||
### Refusing a connection
|
||
|
||
Filters are consulted in registration order, first denial wins:
|
||
|
||
| Filter | Source | Scope |
|
||
|---|---|---|
|
||
| `firewall` | `Configuration/firewall.json`, mutable in-game | Admin-curated, permanent |
|
||
| `blocklist` | `Configuration/ip-blocklist.txt` (millions of entries, opt-in) | Reputation feeds |
|
||
| `auto-denylist` | In-memory, 15 min | What this shard just caught misbehaving |
|
||
|
||
### Contributing a ban
|
||
|
||
`BanChannel.Report(address, ttl, reason)` → `BanExemptions.IsExempt` → if not exempt, fan out to every
|
||
reporter (`crowdsec`, `auto-denylist`).
|
||
|
||
### Allowlists
|
||
|
||
Two, with different authority:
|
||
|
||
| List | Source | Revocable? | Covers |
|
||
|---|---|---|---|
|
||
| `ManualAllowlist` | every `ip-allowlist*.txt` (opt-in) | No — an operator said so | Blocking **and** escalation |
|
||
| `LoginAllowlist` | Earned by authenticating, 90-day TTL | Yes — 10 strikes/hour | Blocking **and** escalation |
|
||
|
||
Both are consulted **only after the blocklist has already matched**, so a normal accept — the one an
|
||
attacker is trying to flood — pays nothing for them. The accept gate itself is deliberately allowlist-free:
|
||
a whitelist there could only turn a deny into an allow at the cost of a lookup on every accept, the
|
||
attacker's included.
|
||
|
||
## Unblocking a player
|
||
|
||
### 1. Find out what is actually blocking them
|
||
|
||
```bash
|
||
# In the reputation blocklist?
|
||
grep -x "203.0.113.42" Distribution/Configuration/ip-blocklist.txt
|
||
|
||
# A live external decision? (this is what survives a restart)
|
||
cscli decisions list --ip 203.0.113.42
|
||
|
||
# Admin-curated?
|
||
grep 203.0.113.42 Distribution/Configuration/firewall.json
|
||
```
|
||
|
||
If none of those match, they may be inside a **CIDR** in the blocklist, or held by the in-memory
|
||
`auto-denylist` — that one is not queryable and expires on its own within 15 minutes.
|
||
|
||
### 2. Add them to the allowlist
|
||
|
||
Set `"enabled": true` in `Configuration/ip-allowlist.json` first — it is off by default, so a shard that
|
||
has never written a carve-out does not poll for one. The shard logs a warning at startup if allowlist files
|
||
are present while the flag is off.
|
||
|
||
Then one entry per line in `Distribution/Configuration/ip-allowlist.txt` — a bare address or a CIDR. This
|
||
file is yours; the generator creates it once and never rewrites it.
|
||
|
||
```
|
||
203.0.113.42 # shard owner, listed via a shared upstream address
|
||
198.51.100.0/24 # a whole range if the ISP rotates within it
|
||
```
|
||
|
||
With the flag on, the shard reloads within `reloadInterval` (60s default). **No restart, and no need to
|
||
re-run the generator.** From that point the address is neither blocked nor contributed. With the flag off
|
||
the generator still subtracts the file at generation time, so the address stops being *blocked* — but a
|
||
behavioural detection can still contribute it, which is the case the flag exists to cover.
|
||
|
||
### 3. Clear any ban that already exists
|
||
|
||
A config change cannot retract a ban that has already left the building:
|
||
|
||
```bash
|
||
cscli decisions delete -i 203.0.113.42
|
||
```
|
||
|
||
### 4. If it keeps coming back
|
||
|
||
The shard is no longer contributing them, so a recurring ban is coming from CrowdSec's own sources (the
|
||
community blocklist, another watcher). Allowlist it there too:
|
||
|
||
```bash
|
||
cscli allowlists create shard-staff -d "Known-good player addresses"
|
||
cscli allowlists add shard-staff 203.0.113.42
|
||
```
|
||
|
||
### What will NOT work
|
||
|
||
- **Deleting the CrowdSec decision alone.** If the address is still in `ip-blocklist.txt` and not
|
||
allowlisted, the next connection re-reports it within `promoteSuppression` (60s).
|
||
- **Editing `ip-blocklist.txt` by hand.** The next generator run rewrites the whole file.
|
||
- **`cscli allowlists` alone.** That is the enforcement layer. The shard's own accept gate sits upstream of
|
||
it and will still refuse the connection.
|
||
|
||
## Why entries appear that should not
|
||
|
||
Reputation feeds list shared consumer address space constantly. On CGNAT one public address fronts many
|
||
subscribers **at the same time**, so a single abusive customer gets the address listed and everyone else
|
||
behind it is blocked with them — and where leases rotate, a listing says little about whoever holds the
|
||
address now. This is near-universal on mobile carriers, satellite (Starlink) and WISPs, and common on
|
||
fixed-line broadband outside North America.
|
||
|
||
A **carve-out** exempts a whole network. **None ship with ModernUO** — which providers to exempt depends on
|
||
where your players actually are, and a carve-out names a real network, so you build the ones you need:
|
||
|
||
```powershell
|
||
# Exempt a CGNAT provider whose players keep getting listed
|
||
.\Export-IpBlocklist.ps1 -AddCarveout starlink -Asn 14593
|
||
|
||
# Later: bring every carve-out up to date with what those networks currently announce
|
||
.\Export-IpBlocklist.ps1 -RefreshCarveouts
|
||
```
|
||
|
||
That writes `ip-allowlist-starlink.txt` beside the blocklist, and every `ip-allowlist*.txt` there is
|
||
subtracted by the generator with no config edit. For the shard to read them too — which is what also stops
|
||
a carve-out address being *contributed* by a behavioural detection — set `enabled` in `ip-allowlist.json`;
|
||
it is off by default so no shard polls for files it never wrote. Starlink costs about 0.1% of the list.
|
||
Blank a file (keep the file) to reputation-block that network again; delete it to drop the carve-out.
|
||
|
||
Carve-out files carry an `asn=` marker in their header, which is how `-RefreshCarveouts` finds them. A
|
||
hand-written allowlist has no marker and is never rewritten.
|
||
|
||
**Do you need one?** If players report being blocked and they are on satellite, mobile, or an ISP short on
|
||
IPv4, probably yes. Find the ASN by looking up an address the network hands out on any public BGP lookup.
|
||
|
||
Prefixes come from **announcements, not ownership records**. Registry data disagrees with what is actually
|
||
routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and
|
||
`206.83.96.0/19` reads as APNIC in RDAP even though `206.83.96/21` is announced by Starlink.
|
||
|
||
## Behavioural detection
|
||
|
||
Verdicts the shard reaches by watching a connection, rather than by consulting a list:
|
||
|
||
| Reason | Trigger |
|
||
|---|---|
|
||
| `rate-limit` | Too many connection attempts in the limiter's window |
|
||
| `silent-connect` | Reaped after `ConnectingSocketIdleLimit` (5s) having sent **zero bytes** |
|
||
| `invalid-seed` | Opened with a zero seed, which no real client sends |
|
||
| `foreign-protocol` | Positively identified as HTTP, TLS or SSH |
|
||
|
||
`BanReasons.IsBehavioral` gates two things: only these may be exempted, and only these enter the
|
||
`auto-denylist`. It is an **opt-in list**, not "everything except `manual`" — a reason added later escalates
|
||
normally rather than silently inheriting an exemption. `manual` is never exempt and never auto-denied.
|
||
|
||
Escalation is **immediate**, on the first detection: a 15-minute local hold plus a `badConnectDuration`
|
||
(4h) contribution. There is no N-connection threshold; the strike counter governs only revoking a
|
||
`LoginAllowlist` entry.
|
||
|
||
The local hold runs 15 minutes from the **first** detection and is never extended by later ones, so an
|
||
address that keeps trying is released on schedule rather than held indefinitely. It does not get a free
|
||
run: the rate limiter sits *ahead* of the connection filters, so a flooder is re-reported and re-held on
|
||
its next attempt. Not refreshing is what keeps the holds in expiry order, which is what makes retiring
|
||
lapsed ones cost the number expiring rather than the number held.
|
||
|
||
### What is deliberately NOT detected
|
||
|
||
**Do not add rules based on arrival framing.** TCP has no message boundaries, so the network, the OS or a
|
||
middlebox can split the opening bytes anywhere regardless of what the client sent. A rule of the form
|
||
"these bytes must arrive together" is broken by construction and drops real players on poor links. This has
|
||
been tried and reverted before.
|
||
|
||
**Do not treat unreadable payloads as hostile.** A legitimate client with encryption enabled when the shard
|
||
expects none sends a structurally perfect connection whose payload is noise — `LoginEncryption.ClientDecrypt`
|
||
is a byte-for-byte stream XOR, so it preserves length exactly while destroying content. This is why
|
||
detection asks "is this positively some *other* protocol?" rather than "is this a good UO client?": however
|
||
misconfigured a UO client is, it never sends `GET / HTTP/1.1`.
|
||
|
||
**Timeouts are keyed on bytes-received, not elapsed time.** A connection that sent *something* and ran out
|
||
of time is far more likely a slow link than an attack. Banning those produces the worst failure mode
|
||
available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being
|
||
firewalled off. Shortening the 5s handshake window has been tried and broke real players.
|
||
|
||
## Known limits
|
||
|
||
- **An allowlist cannot bootstrap.** A `LoginAllowlist` entry is only earned by getting in, so it can never
|
||
repair an existing false positive, and it is weakest on rotating CGNAT — a player whose lease moved is a
|
||
stranger again. `ManualAllowlist` is the fix for that, which is why it is manual — and opt-in, via
|
||
`ip-allowlist.json`.
|
||
- **A never-logged-in player on a shared address can still be caught**, for up to `badConnectDuration`, if a
|
||
co-tenant misbehaves. Accepted: it is 4h and self-healing. The cheapest lever is `badConnectDuration`.
|
||
- **`MaxConnections` (4096) is a hard ceiling.** The accept gate runs *after* the kernel completed the TCP
|
||
handshake, so a blocklist match saves the socket setup and the `NetState` slot but never the connection
|
||
itself. Only an upstream L4 proxy or edge scrubbing moves that cost off the shard.
|
||
- **The `auto-denylist` stops tracking at `maxEntries`.** Past it a detection still disconnects the
|
||
connection, but the address is not held, so it pays full detection cost on every reconnect instead of a
|
||
cheap accept-gate deny. The default is sized for the 50k–250k distinct-source floods seen in practice; a
|
||
flood past it wants upstream scrubbing rather than a larger cap, which only buys a longer on-loop scan.
|
||
|
||
## Configuration
|
||
|
||
| File | Controls |
|
||
|---|---|
|
||
| `bans.json` | `reportRateLimitTrips`, `autoBanDuration`, `reportBadConnects`, `badConnectDuration` |
|
||
| `blocklist.json` | `enabled` (default `false`), `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` |
|
||
| `ip-allowlist.json` | `enabled` (default `false`), `files` (wildcards allowed), `reloadInterval` |
|
||
| `login-allowlist.json` | `enabled`, `file`, `ttl`, `flushInterval`, `escalateAfterStrikes`, `strikeWindow` |
|
||
| `auto-denylist.json` | `enabled`, `duration`, `maxEntries` (default `324,449` — sized for the floods seen in practice; see the remark on the setting before raising it) |
|
||
| `crowdsec.json` | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` |
|
||
| `firewall.json` | Admin-curated entries |
|
||
|
||
A shard fronted by an upstream proxy can disable all of it and register nothing.
|
||
|
||
## Key files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `Projects/Server/Network/IConnectionFilter.cs` | Accept-path gate contract |
|
||
| `Projects/Server/Network/ConnectionFilters.cs` | Filter registry + lifecycle |
|
||
| `Projects/Server/Network/ForeignProtocol.cs` | Positive identification of non-UO traffic |
|
||
| `Projects/Server/Network/Bans/BanChannel.cs` | Contribution fan-out + `IsExempt` seam |
|
||
| `Projects/Server/Network/Bans/BanReasons.cs` | Reason slugs + the behavioural opt-in set |
|
||
| `Projects/UOContent/Network/BanExemptions.cs` | Combines both allowlists into one answer |
|
||
| `Projects/UOContent/Network/Blocklist/BlocklistFilter.cs` | File-sourced blocklist filter |
|
||
| `Projects/UOContent/Network/Blocklist/ManualAllowlist.cs` | Operator carve-outs, read from the allowlist files |
|
||
| `Projects/UOContent/Network/LoginAllowlist/LoginAllowlist.cs` | Allowlist earned by authenticating |
|
||
| `Projects/UOContent/Network/AutoDenylist/AutoDenylist.cs` | Short-lived local hold |
|
||
| `Projects/UOContent/Network/CrowdSec/CrowdSecReporter.cs` | LAPI contribution sink |
|
||
| `Projects/UOContent/Network/Firewall/Firewall.cs` | Admin-curated firewall set |
|
||
| `tools/Export-IpBlocklist.ps1` | Blocklist generator + allowlist subtraction |
|