5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kamron Batman
|
6dcb59e00b
|
refactor: drop the deprecated allowlistFiles key from blocklist config
Blocklists have not been out long enough for anyone to have set "allowlistFiles" in blocklist.json, so the compatibility shim was protecting nobody. Removes the bound property, ManualAllowlist's WarnOnDeprecatedKey, and the test pinning the null default. The key now binds nowhere. JsonConfig ignores unknown keys, so a config that still carries it loads fine and the setting is simply inert. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kamron Batman
|
bd187bd252
|
refactor: rename FileAllowlist to ManualAllowlist
"File" described the storage, which is the least interesting thing about it. The distinction from LoginAllowlist is provenance: one is declared by an operator, the other is earned by authenticating. Both are IP allowlists, so IpAllowlist would not have separated them either. "Manual" matches vocabulary already in the subsystem -- BanReasons.Manual is the operator-declared ban, and the docs already describe this list as "an operator said so". Also moves both files into Network/ManualAllowlist/, mirroring Network/LoginAllowlist/. They no longer belong under Blocklist/ now that the config is decoupled and BanExemptions is the consumer that cannot be served any other way. Namespace is unchanged, so no using directives move. Configuration/ip-allowlist.json keeps its name: it is named for the ip-allowlist*.txt files it governs, whose names the generator owns, and that grouping is what an operator browsing Configuration/ actually sees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kamron Batman
|
b8b5a1e0c9
|
docs: tighten the comments across the ban subsystem
Comments only -- no code changed. Covers the blocklist, file allowlist, login allowlist, auto-denylist, CrowdSec reporter and firewall. Removes development narration: "(Task 2)" left over from a plan in Firewall.PurgeExpired, "matching the per-feature JSON config pattern used by X" repeated across four config loaders, "Recurring and tokenless before" describing an edit rather than the code, and a duplicated threading note in BlocklistSnapshot that the inline comment already made. Points Firewall at dev-docs/ip-bans-and-allowlists.md instead of a "ban-channel design doc" that does not exist. Fixes a stale reference the cleanup surfaced: FileAllowlist's remarks still told readers to opt in via blocklist.json's "allowlistEnabled", which moved to "enabled" in ip-allowlist.json. Kept, in tightened form, everything that encodes a reason the code cannot show: the CrowdSec culture and LAPI protocol notes, why the reporter flushes off the loop thread at shutdown, why FullMode.Wait rather than DropWrite, why Firewall reads Core.Now instead of DateTime.UtcNow, and the volatile-swap contracts on both snapshots. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kamron Batman
|
19b8ce2ce4
|
feat: make the blocklist and file allowlist opt-in, decouple their config
Both features ran on every shard out of the box, each polling on its own 60s timer for files most shards never generate. Neither was ever asked for. Blocklist: the only disable path was an empty "file", and the default is non-empty, so Start() always reached Task.Run(PollLoop) plus a recurring SweepGuard timer. Adds "enabled" (default false). A shard that has a list on disk but no "enabled" key logs a Warning rather than silently dropping a gate it was relying on. File allowlist: moves out of blocklist.json into its own ip-allowlist.json with "enabled" (default false), "files" and "reloadInterval". It was never a sub-feature of the blocklist -- its two consumers are BlocklistFilter, where the generator already subtracts these files anyway, and BanExemptions, which suppresses behavioural ban contributions and works on a shard running no blocklist at all. That second consumer is the only mechanism for what it does, so a shared flag could not express it. "allowlistFiles" stays bound on BlocklistSettings, defaulting to null and deliberately not honoured, purely so an operator who set it is told where it went instead of losing the carve-out silently. The two still work together: the blocklist warns at startup when it is on and the file allowlist is not, since only the generator's subtraction is covering carve-outs then, and that does not cover ban contributions. Also fixes the promote-guard sweep, which was recurring and tokenless, so Stop() cancelled the poll but left the sweep running and a later Start() added another. It is now held and stopped, and only started when hits are reported -- nothing can mark the guard otherwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kamron Batman
|
aae173a797
|
feat(network): allowlist false-positive IPs, escalate on behavior (#2556)
## Why The shard owner, on a Starlink CGNAT address, was blocked by the imported reputation blocklist. The cause was not CrowdSec. The address was a literal line in `ip-blocklist.txt`, so `BlocklistFilter` denied it at accept and then promoted it — and clearing the CrowdSec decision could not fix it either, because the file entry re-reports within `promoteSuppression` of every reconnect attempt. This is structural, not a one-off. Reputation feeds list shared consumer address space constantly: on CGNAT one public address fronts many subscribers **at the same time**, so a single abusive customer gets the address listed and everyone else behind it is blocked with them. Where leases rotate, a listing says little about whoever holds the address now. Around 1,000 Starlink addresses sit in the current list. So exemptions go where they cost nothing, and escalation is driven by what a connection actually does. ## Generator — `tools/Export-IpBlocklist.ps1` `-AllowlistFile` takes multiple paths, subtracted from the merged set before the output is written. Defaults to every `ip-allowlist*.txt` beside the output, merged into one allow set: - `ip-allowlist.txt` — operator exemptions, created once and **never rewritten** - `ip-allowlist-<name>.txt` — a carve-out you built, regenerable and copyable between shards **Subtraction is range-correct.** An allowlisted address inside a blocked CIDR splits that CIDR around the hole rather than being silently ignored. This also fixes `-ExcludeAnonymizers`, which parsed CIDR entries into `$anonCidr` and then only ever subtracted singles. **No carve-out ships.** A carve-out names a real network, and which ones a shard should exempt depends on where its players actually are — so publishing one would make that policy call for every shard and put a specific provider's address space in the repo. The script builds them on request instead: ```powershell .\Export-IpBlocklist.ps1 -AddCarveout starlink -Asn 14593 ``` Carve-outs are **discovered, not configured**: every `ip-allowlist*.txt` beside the output is subtracted, by the generator and by the shard, so a file an admin adds needs no config edit and no code change. Each carries an `asn=` marker in its header, which is how `-RefreshCarveouts` rebuilds it without the script keeping a list of anyone's networks; a hand-written allowlist has no marker and is never rewritten. Prefixes come from **announcements, not ownership records**, because registry data disagrees with what is actually routed and silently caps result sets: ARIN whois returns at most 256 rows and gives per-customer /24s, and `206.83.96.0/19` reads as APNIC in RDAP even though `206.83.96/21` is announced by Starlink. Editing an allowlist bypasses `-MinInterval`, so a just-added exemption isn't indistinguishable from the allowlist not working. A Starlink carve-out, if you build one, costs **~4,300 IPs + ~144 CIDRs of 4.2M (0.10%)**. ## Allowlists **`FileAllowlist`** reads the same files the generator subtracts, so an operator entry means "leave this address alone" for real. Subtraction alone only covers being *blocked*; behavioural detections never consult the blocklist, so without this a carve-out was quietly routed around — one scanner behind a shared address was enough to get everyone behind it contributed and firewalled, with nothing in the shard's own config explaining why. Reading the files also means an entry applies on the next reload rather than the next regeneration, which is what matters when someone is complaining now. **`LoginAllowlist`** is earned by authenticating, with a 90-day TTL because an address that logged in years ago is a stranger. Its own store rather than `Account.LoginIPs`, which has no timestamps and cannot be backfilled. An entry is evidence rather than a licence: 10 suppressed contributions in an hour revokes it, and a fresh login forgives the tally. Both are consulted **only after the blocklist has already matched**, so a normal accept pays nothing for them and the accept gate stays allowlist-free. `BanExemptions` combines them behind `BanChannel.IsExempt` and suppresses escalation only — every local defence still applies. Two limits, both deliberate and documented in the class: `LoginAllowlist` **cannot bootstrap** (an entry is only earned by getting in, so it never repairs an existing false positive), and it is weakest on rotating CGNAT. That is why `FileAllowlist` is the fix for those, and why it is manual. ## Behavioural detection | Reason | Trigger | |---|---| | `silent-connect` | Reaped after 5s having sent **zero bytes** | | `invalid-seed` | Opened with a zero seed | | `foreign-protocol` | Positively identified as HTTP, TLS or SSH | **`ForeignProtocol` inverts the test.** Asking "is this a good UO client?" cannot work: `LoginEncryption.ClientDecrypt` is a byte-for-byte stream XOR, so a legitimate client with encryption enabled when the shard expects none sends a structurally perfect connection whose payload is noise. "Speaks HTTP" is safe where "unreadable" is not — however misconfigured a UO client is, it never sends `GET / HTTP/1.1`. Nothing assumes arrival framing. TCP has no message boundaries, so a rule of the form "these bytes must arrive together" is broken by construction and drops real players on poor links. A prefix match with too few bytes to confirm waits for more. A four-byte seed can legitimately spell `GET ` (the address 71.69.84.32) or `0x16 0x03 0x0?` (22.3.x.x), so confirmation requires the request line to continue in printable ASCII or an actual ClientHello inside a plausible record — a real client's fifth byte is a packet id (`0x80`, `0x91`, `0xEF`), none of them printable, so those collisions fall through. Everything is keyed on **bytes-received rather than elapsed time**. A connection that sent something and ran out of time is far more likely a slow link than an attack, and banning those produces the worst failure mode available: the player retries, trips the rate limiter, and compounds a bad connection into hours of being firewalled off. ## `AutoDenylist` A short-lived local hold (15m) on behavioural detections, as `IConnectionFilter` + `IBanReporter` over one store so the engine detection sites never reach into content. This closes the gap where a flood pays for a socket, buffer and `NetState` slot per connection while waiting for the OS bouncer — the verdicts that matter most are reachable only *after* reading bytes — and it is the entire defence on a shard running no bouncer, which is the default config. Not persisted: a holding pen that survives restarts is a ban without a ban's review. Cost: one dictionary lookup on a usually-empty dict per accept. ## `BanReasons` Centralises the reason slugs. `IsBehavioral` is an **opt-in** set, not "everything except manual", so a future reason escalates normally instead of silently inheriting an exemption or entering a local denylist. This caught a real bug during review: the first cut of the exemption swallowed `manual` admin bans (`Commands.cs`, three sites in `AdminGump`) for any allowlisted address. ## Fixes found in review - **`BanConfiguration.Settings` was null until `Configure()` ran**, while the reap path dereferences it every `Slice()`. A harness driving `NetState.Slice()` directly hit an NRE that presented as flaky because it depended on whether an earlier test had already called `Configure()` — which is why it failed on some CI platforms and not others. Now starts at the record's defaults, with idempotency tracked by a flag; this also removes the same latent NRE from the pre-existing rate-limit path. - **`-AllowlistFile` was typed `[string]`** while documented and used as a list, so passing two paths would have collapsed them into one string. ## Layout and docs Content network code moves out of `Misc/` into `UOContent/Network/`, one concern per folder — `AutoDenylist/`, `Blocklist/`, `CrowdSec/`, `Firewall/`, `LoginAllowlist/`, `Packets/`. **Namespaces are untouched**, so these are pure file moves (git tracks all 16 as renames). `dev-docs/ip-bans-and-allowlists.md` documents the subsystem, leading with the operator process for unblocking a player — including the three things that look sufficient and are not: deleting the CrowdSec decision alone, editing `ip-blocklist.txt` by hand, and `cscli allowlists` alone. `.gitignore` covers the new config files. ## Testing Build clean. **Server.Tests 810 passed**, **UOContent.Tests 637 passed**, zero warnings. This branch adds 38 tests; the rest of the delta is main's, since this is rebased on current `main`. New coverage: TTL boundary and renewal, private-address exclusion, manual-ban-never-exempt, unopted-reason-never-exempt, strike revocation, quiet-window reset, login forgiveness, file-allowlist CIDR coverage, file-allowlist not spending the earned list's strikes, denylist expiry-on-read, cap enforcement, lapsed-entry reclaim, HTTP/TLS/SSH identification, seed-collision fall-through, and encrypted-login-is-not-foreign. Generator verified end-to-end against live feeds: a clean run ships no carve-out, `-AddCarveout starlink -Asn 14593` fetches and collapses 213 prefixes to 115 ranges in 0.1s over 4.2M entries, `-RefreshCarveouts` rediscovers it by its `asn=` marker, a hand-written allowlist is left untouched, and deleting a carve-out drops it rather than having it rewritten. CIDR splitting verified exhaustively: a single-IP hole in a /24 leaves exactly 255 of 256 addresses blocked. ## Operator note Existing installs are unaffected until the generator next runs, which creates `ip-allowlist.txt` and nothing else. To unblock someone: add the address to that file and delete any live CrowdSec decision — the existing ban outlives the config change. The shard picks the entry up on its next reload, so re-running the generator is optional. A shard whose players are on CGNAT (satellite, mobile, or an ISP short on IPv4) will likely also want `-AddCarveout`; see `dev-docs/ip-bans-and-allowlists.md`. ## Also included: a latent CI failure this PR surfaced `fix(tests): serialize test classes that rent through STArrayPool` touches a property-list test file that has nothing to do with this feature. It is here because it was failing macOS CI, and it is trivially cherry-pickable out if you would rather it went to `main` on its own — **which may be the better call, since it is failing `main` today.** CI has since gone green with it applied. `STArrayPool` is single-threaded by design and its bucket cache is a plain `static`, not `[ThreadStatic]`, with a check-then-act initialize in `Return()`: ```csharp var cacheBuckets = _cacheBuckets ?? InitializeBuckets(); ``` Two threads both see null, both initialize, and the loser trips `Debug.Assert(_cacheBuckets is null)`. Anything renting from it has to stay off parallel test threads — which is what the `DisableParallelization` collections are for. - `ObjectPropertyListReentrancyTests` and `ObjectPropertyListNestedBuildTests` (added in #2555) build property lists, which rent the interpolation buffer, but were not in the sequential collection — unlike `PropertyListInvalidationDuringBuildTests` in the same file. This is a **latent failure already on `main`**; it is timing-dependent, so it shows on some platforms and not others. - `AutoDenylistTests` (added here) has the same exposure: its cap tests reach `AutoDenylist.Sweep`, which rents a `PooledRefList` without `mt`. The blocklist tests need no marking because `BlocklistSnapshot.Build` asks for the `mt` pool explicitly. No production change — `STArrayPool` is the right pool on the game loop, where both `Sweep` and the property list actually run. ## Deliberately not included Waiting for a fragmented four-byte seed at `AwaitingSeed`. It looked like a bug but the disconnect is a deliberate defence: only pre-0xEF clients reach it (0xEF goes through `HandlePacket`, which already waits for its 21 bytes), and waiting converts an instant drop into a full 5s slot hold for a client sending one or two bytes, or a loris dribbling a byte every few seconds. Against a fixed 4096-entry `MaxConnections` table that trades capacity that matters for a fragmentation case a reconnect already fixes. |
Renamed from Projects/UOContent/Misc/Blocklist/BlocklistConfiguration.cs (Browse further)