feat(network): allowlist false-positive IPs, escalate on behaviour

The shard owner, on a Starlink CGNAT address, was blocked by the imported
reputation blocklist. The cause was not CrowdSec: the address was a literal line
in ip-blocklist.txt, so BlocklistFilter denied it at accept and then promoted it.
Clearing the CrowdSec decision could not fix it either, because the file entry
re-reports within promoteSuppression of every reconnect.

Reputation feeds list shared consumer address space constantly. On CGNAT one
public address fronts many subscribers at the same time, so a single abusive
customer gets the address listed and everyone else behind it is blocked with
them; where leases rotate, a listing says little about whoever holds the address
now. So exemptions go where they cost nothing, and escalation is driven by what a
connection actually does.

GENERATOR (tools/Export-IpBlocklist.ps1)

- -AllowlistFile takes multiple paths, subtracted from the merged set before the
  output is written. Defaults to the operator's own ip-allowlist.txt (created
  once, never rewritten) plus one generated file per network carve-out.
- Subtraction is range-correct: an allowlisted address inside a blocked CIDR
  splits that CIDR around the hole instead of being ignored. This also fixes
  -ExcludeAnonymizers, which parsed CIDR entries and then only subtracted
  singles.
- Carve-outs are a table (name, ASN, reason, offline seed) rather than a
  hardcoded network, so covering another CGNAT provider is one row.
  -RefreshCarveouts re-fetches from the ASN's current routing announcements and
  collapses them; -Carveout '' subtracts none. Announcements rather than
  ownership records, because registry data disagrees with what is routed and caps
  its result sets.
- Editing an allowlist bypasses -MinInterval, so a just-added exemption is not
  indistinguishable from the allowlist not working.
- The shipped starlink carve-out costs ~0.1% of the list.

ALLOWLISTS

- FileAllowlist reads the same files the generator subtracts, so an operator
  entry means "leave this address alone" for real. Subtraction alone only covers
  being BLOCKED; behavioural detections never consult the blocklist, so without
  this a carve-out was quietly routed around and one scanner behind a shared
  address was enough to get everyone behind it firewalled. Reading the files also
  means an entry applies on the next reload rather than the next regeneration.
- LoginAllowlist is earned by authenticating, with a 90 day TTL because an
  address that logged in years ago is a stranger. Its own store rather than
  Account.LoginIPs, which has no timestamps and cannot be backfilled. An entry is
  evidence rather than a licence: 10 suppressed contributions in an hour revokes
  it, and a fresh login forgives the tally.
- Both are consulted only AFTER the blocklist has already matched, so a normal
  accept pays nothing for them and the accept gate stays allowlist-free.
- BanExemptions combines them behind BanChannel.IsExempt. Suppresses escalation
  only; every local defence still applies.

BEHAVIOURAL DETECTION

- silent-connect (reaped having sent zero bytes) and invalid-seed (a zero seed)
  are contributed. Both were already disconnected.
- ForeignProtocol positively identifies HTTP, TLS and SSH. Asking "is this a good
  UO client?" cannot work: LoginEncryption.ClientDecrypt is a byte-for-byte
  stream XOR, so a client with encryption on when the shard expects none sends a
  structurally perfect connection whose payload is noise. Nothing assumes arrival
  framing, since TCP has no message boundaries and a rule of the form "these
  bytes must arrive together" drops real players on poor links.
- Keyed on bytes-received rather than elapsed time throughout. A connection that
  sent something and ran out of time is far more likely a slow link, and banning
  those makes the player retry, trip the rate limiter, and compound it.
- AutoDenylist holds behavioural detections locally for 15 minutes, as an
  IConnectionFilter plus IBanReporter over one store so engine detection sites
  never reach into content. Closes the gap where a flood pays for a socket and a
  NetState per connection while waiting for an OS bouncer, and is the whole
  defence on a shard running none. Not persisted: a holding pen that survives
  restarts is a ban without a ban's review.
- BanReasons centralises the slugs. IsBehavioral is an opt-in set, not
  "everything except manual", so a future reason escalates normally instead of
  silently inheriting an exemption. The first cut of the exemption swallowed
  manual admin bans; this is why.

FIXES

- BanConfiguration.Settings was null until Configure() ran while the reap path
  dereferences it every Slice(), so a harness driving NetState.Slice() directly
  hit an NRE that looked flaky because it depended on test ordering.
- -AllowlistFile was typed [string] while documented and used as a list.

LAYOUT AND DOCS

Content network code moves out of Misc into UOContent/Network, one concern per
folder. Namespaces are untouched, so these are pure file moves.
dev-docs/ip-bans-and-allowlists.md documents the subsystem, leading with the
operator process for unblocking a player -- including the three things that look
sufficient and are not: deleting the CrowdSec decision alone, editing
ip-blocklist.txt by hand, and cscli allowlists alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-30 18:31:34 -07:00

10 KiB

Raw Blame History

IP Bans, Blocklists and Allowlists

How a shard decides to refuse a connection, how it contributes bans to an external bouncer, and how an operator exempts someone who was caught by mistake.

If you are here because a player cannot connect, skip to Unblocking a player.

The shape of it

Two independent questions, deliberately separated:

Question	Answered by	Effect
Refuse this connection?	`IConnectionFilter` implementations, at accept	The socket is dropped
Tell the outside world about it?	`BanChannel` → `IBanReporter` implementations	CrowdSec, and from there an OS bouncer

BanChannel never enforces and filters never report on each other's behalf. Enforcement that outlives the process belongs to the OS bouncer; the shard only contributes.

Refusing a connection

Filters are consulted in registration order, first denial wins:

Filter	Source	Scope
`firewall`	`Configuration/firewall.json`, mutable in-game	Admin-curated, permanent
`blocklist`	`Configuration/ip-blocklist.txt` (millions of entries)	Reputation feeds
`auto-denylist`	In-memory, 15 min	What this shard just caught misbehaving

Contributing a ban

BanChannel.Report(address, ttl, reason) → BanExemptions.IsExempt → if not exempt, fan out to every reporter (crowdsec, auto-denylist).

Allowlists

Two, with different authority:

List	Source	Revocable?	Covers
`FileAllowlist`	`ip-allowlist.txt`, `ip-allowlist-<carveout>.txt`	No — an operator said so	Blocking and escalation
`LoginAllowlist`	Earned by authenticating, 90-day TTL	Yes — 10 strikes/hour	Blocking and escalation

Both are consulted only after the blocklist has already matched, so a normal accept — the one an attacker is trying to flood — pays nothing for them. The accept gate itself is deliberately allowlist-free: a whitelist there could only turn a deny into an allow at the cost of a lookup on every accept, the attacker's included.

Unblocking a player

1. Find out what is actually blocking them

# In the reputation blocklist?
grep -x "203.0.113.42" Distribution/Configuration/ip-blocklist.txt

# A live external decision? (this is what survives a restart)
cscli decisions list --ip 203.0.113.42

# Admin-curated?
grep 203.0.113.42 Distribution/Configuration/firewall.json

If none of those match, they may be inside a CIDR in the blocklist, or held by the in-memory auto-denylist — that one is not queryable and expires on its own within 15 minutes.

2. Add them to the allowlist

One entry per line in Distribution/Configuration/ip-allowlist.txt — a bare address or a CIDR. This file is yours; the generator creates it once and never rewrites it.

203.0.113.42        # shard owner, listed via a shared upstream address
198.51.100.0/24     # a whole range if the ISP rotates within it

The shard reloads within reloadInterval (60s default). No restart, and no need to re-run the generator. From that point the address is neither blocked nor contributed.

3. Clear any ban that already exists

A config change cannot retract a ban that has already left the building:

cscli decisions delete -i 203.0.113.42

4. If it keeps coming back

The shard is no longer contributing them, so a recurring ban is coming from CrowdSec's own sources (the community blocklist, another watcher). Allowlist it there too:

cscli allowlists create shard-staff -d "Known-good player addresses"
cscli allowlists add shard-staff 203.0.113.42

What will NOT work

Deleting the CrowdSec decision alone. If the address is still in ip-blocklist.txt and not allowlisted, the next connection re-reports it within promoteSuppression (60s).
Editing ip-blocklist.txt by hand. The next generator run rewrites the whole file.
cscli allowlists alone. That is the enforcement layer. The shard's own accept gate sits upstream of it and will still refuse the connection.

Why entries appear that should not

Reputation feeds list shared consumer address space constantly. On CGNAT one public address fronts many subscribers at the same time, so a single abusive customer gets the address listed and everyone else behind it is blocked with them — and where leases rotate, a listing says little about whoever holds the address now. This is near-universal on mobile carriers, satellite (Starlink) and WISPs, and common on fixed-line broadband outside North America.

The generator therefore keeps a carve-out table, one entry per provider, each written to its own ip-allowlist-<name>.txt and subtracted from the output. starlink ships active, costing about 0.1% of the list. Blank a file (keep the file) to reputation-block that network again; deleting it makes the next run write it back.

# Bring carve-out data up to date with what each network currently announces
.\Export-IpBlocklist.ps1 -RefreshCarveouts

# Generate with no carve-out at all, keeping your own allowlist
.\Export-IpBlocklist.ps1 -Carveout ''

Covering another CGNAT provider is a row in the $Carveouts table near the top of the script: a name, its ASN, a one-line reason, and either pasted prefixes or nothing at all if you intend to use -RefreshCarveouts.

Prefixes come from announcements, not ownership records. Registry data disagrees with what is actually routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and 206.83.96.0/19 reads as APNIC in RDAP even though 206.83.96/21 is announced by Starlink.

Behavioural detection

Verdicts the shard reaches by watching a connection, rather than by consulting a list:

Reason	Trigger
`rate-limit`	Too many connection attempts in the limiter's window
`silent-connect`	Reaped after `ConnectingSocketIdleLimit` (5s) having sent zero bytes
`invalid-seed`	Opened with a zero seed, which no real client sends
`foreign-protocol`	Positively identified as HTTP, TLS or SSH

BanReasons.IsBehavioral gates two things: only these may be exempted, and only these enter the auto-denylist. It is an opt-in list, not "everything except manual" — a reason added later escalates normally rather than silently inheriting an exemption. manual is never exempt and never auto-denied.

Escalation is immediate, on the first detection: a 15-minute local hold plus a badConnectDuration (4h) contribution. There is no N-connection threshold; the strike counter governs only revoking a LoginAllowlist entry.

What is deliberately NOT detected

Do not add rules based on arrival framing. TCP has no message boundaries, so the network, the OS or a middlebox can split the opening bytes anywhere regardless of what the client sent. A rule of the form "these bytes must arrive together" is broken by construction and drops real players on poor links. This has been tried and reverted before.

Do not treat unreadable payloads as hostile. A legitimate client with encryption enabled when the shard expects none sends a structurally perfect connection whose payload is noise — LoginEncryption.ClientDecrypt is a byte-for-byte stream XOR, so it preserves length exactly while destroying content. This is why detection asks "is this positively some other protocol?" rather than "is this a good UO client?": however misconfigured a UO client is, it never sends GET / HTTP/1.1.

Timeouts are keyed on bytes-received, not elapsed time. A connection that sent something and ran out of time is far more likely a slow link than an attack. Banning those produces the worst failure mode available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being firewalled off. Shortening the 5s handshake window has been tried and broke real players.

Known limits

An allowlist cannot bootstrap. A LoginAllowlist entry is only earned by getting in, so it can never repair an existing false positive, and it is weakest on rotating CGNAT — a player whose lease moved is a stranger again. FileAllowlist is the fix for that, which is why it is manual.
A never-logged-in player on a shared address can still be caught, for up to badConnectDuration, if a co-tenant misbehaves. Accepted: it is 4h and self-healing. The cheapest lever is badConnectDuration.
MaxConnections (4096) is a hard ceiling. The accept gate runs after the kernel completed the TCP handshake, so a blocklist match saves the socket setup and the NetState slot but never the connection itself. Only an upstream L4 proxy or edge scrubbing moves that cost off the shard.

Configuration

File	Controls
`bans.json`	`reportRateLimitTrips`, `autoBanDuration`, `reportBadConnects`, `badConnectDuration`
`blocklist.json`	`file`, `allowlistFiles`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression`
`login-allowlist.json`	`enabled`, `file`, `ttl`, `flushInterval`, `escalateAfterStrikes`, `strikeWindow`
`auto-denylist.json`	`enabled`, `duration`, `maxEntries`
`crowdsec.json`	`lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue`
`firewall.json`	Admin-curated entries

A shard fronted by an upstream proxy can disable all of it and register nothing.

Key files

File	Role
`Projects/Server/Network/IConnectionFilter.cs`	Accept-path gate contract
`Projects/Server/Network/ConnectionFilters.cs`	Filter registry + lifecycle
`Projects/Server/Network/ForeignProtocol.cs`	Positive identification of non-UO traffic
`Projects/Server/Network/Bans/BanChannel.cs`	Contribution fan-out + `IsExempt` seam
`Projects/Server/Network/Bans/BanReasons.cs`	Reason slugs + the behavioural opt-in set
`Projects/UOContent/Network/BanExemptions.cs`	Combines both allowlists into one answer
`Projects/UOContent/Network/Blocklist/BlocklistFilter.cs`	File-sourced blocklist filter
`Projects/UOContent/Network/Blocklist/FileAllowlist.cs`	Operator carve-outs, read from the allowlist files
`Projects/UOContent/Network/LoginAllowlist/LoginAllowlist.cs`	Allowlist earned by authenticating
`Projects/UOContent/Network/AutoDenylist/AutoDenylist.cs`	Short-lived local hold
`Projects/UOContent/Network/CrowdSec/CrowdSecReporter.cs`	LAPI contribution sink
`Projects/UOContent/Network/Firewall/Firewall.cs`	Admin-curated firewall set
`tools/Export-IpBlocklist.ps1`	Blocklist generator + allowlist subtraction

10 KiB Raw Blame History