feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542)

Reshapes IP banning around one idea: **core owns the question, content owns every answer.** Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter. ## The seam ```csharp public interface IConnectionFilter { string Name { get; } void Configure(); void Start(CancellationToken token); void Stop(); bool ShouldDeny(IPAddress address); } ``` The accept path went from hardcoded branches to one question: ```csharp else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy)) { logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy); } ``` Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS. Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's. A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either. This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic. ## What ships behind it **`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands. **`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall. The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up. The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves. **`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream. ## CrowdSec: contribute-first `IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`. Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work. ### Why not pull decisions from CrowdSec? The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce. The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline. ## Threading policy `CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem: - Anything touching game state runs **only** on the main loop. - Heavy work that *needs* game state must be **chunked** across ticks, never threaded. - Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**. Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it. ## Shared primitives `SortedRangeIndex<T> where T : IBinaryInteger<T>` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it. Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests. `IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated. ## Config | File | Owner | Keys | |---|---|---| | `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` | | `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` | | `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` | | `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) | Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change. ## Notes for review - **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex<UInt128>`, same semantics and public API. - **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`. - **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core. - **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel. - `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR. - The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message. ## Tests **1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader.
2026-07-25 11:59:37 -07:00 · 2026-07-25 11:59:37 -07:00 · c39454137e
commit c39454137e
parent bec4cfa910
52 changed files with 4655 additions and 547 deletions
--- a/Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs
+++ b/Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs
@ -0,0 +1,137 @@
+/*************************************************************************
+ * ModernUO                                                              *
+ * Copyright 2019-2026 - ModernUO Development Team                       *
+ * Email: hi@modernuo.com                                                *
+ * File: CrowdSecAlertClient.cs                                          *
+ *                                                                       *
+ * This program is free software: you can redistribute it and/or modify  *
+ * it under the terms of the GNU General Public License as published by  *
+ * the Free Software Foundation, either version 3 of the License, or     *
+ * (at your option) any later version.                                   *
+ *                                                                       *
+ * You should have received a copy of the GNU General Public License     *
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>. *
+ *************************************************************************/
+
+using System;
+using System.Collections.Generic;
+using System.Net;
+using System.Net.Http;
+using System.Net.Http.Headers;
+using System.Net.Http.Json;
+using System.Text.Json;
+using System.Threading;
+using System.Threading.Tasks;
+
+namespace Server.Network.Bans.CrowdSec;
+
+/// <summary>Reporter-side LAPI operations, mockable for tests.</summary>
+public interface ICrowdSecAlertClient : IDisposable
+{
+    ValueTask PostAlertsAsync(IReadOnlyList<CrowdSecAlert> alerts, CancellationToken token);
+    ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token);
+}
+
+/// <summary>
+/// CrowdSec LAPI watcher client: authenticates with machine credentials and posts/deletes decisions.
+/// Holds a JWT refreshed on expiry or a 401.
+/// </summary>
+public sealed class CrowdSecAlertClient : ICrowdSecAlertClient
+{
+    private static readonly JsonSerializerOptions _jsonOptions = new() { PropertyNameCaseInsensitive = true };
+
+    private readonly HttpClient _http;
+    private readonly string _machineId;
+    private readonly string _password;
+
+    private string _token;
+    private DateTime _tokenExpiresUtc = DateTime.MinValue;
+
+    public CrowdSecAlertClient(CrowdSecSettings settings)
+    {
+        var baseUri = new Uri(settings.LapiUrl, UriKind.Absolute); // fails loud on malformed url
+        _http = new HttpClient { BaseAddress = baseUri, Timeout = TimeSpan.FromSeconds(30) };
+        _http.DefaultRequestHeaders.Add("User-Agent", "ModernUO-watcher/1.0");
+        _machineId = settings.MachineId;
+        _password = settings.Password;
+    }
+
+    private async Task EnsureAuthAsync(CancellationToken token)
+    {
+        if (_token != null && DateTime.UtcNow < _tokenExpiresUtc - TimeSpan.FromMinutes(1))
+        {
+            return;
+        }
+
+        var request = new CrowdSecLoginRequest { MachineId = _machineId, Password = _password };
+        using var response = await _http.PostAsJsonAsync("/v1/watchers/login", request, _jsonOptions, token)
+            .ConfigureAwait(false);
+        response.EnsureSuccessStatusCode();
+
+        var login = await response.Content.ReadFromJsonAsync<CrowdSecLoginResponse>(_jsonOptions, token)
+            .ConfigureAwait(false);
+        _token = login?.Token ?? throw new InvalidOperationException("CrowdSec login returned no token.");
+        _tokenExpiresUtc = DateTime.TryParse(login.Expire, out var exp) ? exp.ToUniversalTime() : DateTime.UtcNow.AddHours(1);
+    }
+
+    private void Authorize(HttpRequestMessage message) =>
+        message.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _token);
+
+    public async ValueTask PostAlertsAsync(IReadOnlyList<CrowdSecAlert> alerts, CancellationToken token)
+    {
+        if (alerts.Count == 0)
+        {
+            return;
+        }
+
+        await SendWithRetryAsync(() =>
+        {
+            var message = new HttpRequestMessage(HttpMethod.Post, "/v1/alerts")
+            {
+                Content = JsonContent.Create(alerts, options: _jsonOptions)
+            };
+            Authorize(message);
+            return message;
+        }, token).ConfigureAwait(false);
+    }
+
+    public async ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token)
+    {
+        var query = BuildDeleteQuery(origin, ip);
+        await SendWithRetryAsync(() =>
+        {
+            var message = new HttpRequestMessage(HttpMethod.Delete, query);
+            Authorize(message);
+            return message;
+        }, token).ConfigureAwait(false);
+    }
+
+    /// <summary>
+    /// Builds the decisions-delete query string. <paramref name="origin"/> is operator-controlled config
+    /// (crowdsec.json), so it must be escaped like any other untrusted value going into a URL.
+    /// </summary>
+    internal static string BuildDeleteQuery(string origin, IPAddress ip) =>
+        $"/v1/decisions?origin={Uri.EscapeDataString(origin)}&ip={Uri.EscapeDataString(ip.ToString())}";
+
+    private async ValueTask SendWithRetryAsync(Func<HttpRequestMessage> build, CancellationToken token)
+    {
+        await EnsureAuthAsync(token).ConfigureAwait(false);
+
+        using var first = build();
+        using var response = await _http.SendAsync(first, token).ConfigureAwait(false);
+        if (response.StatusCode != HttpStatusCode.Unauthorized)
+        {
+            response.EnsureSuccessStatusCode();
+            return;
+        }
+
+        // Token rejected mid-flight: force a re-login and retry once.
+        _token = null;
+        await EnsureAuthAsync(token).ConfigureAwait(false);
+        using var retry = build();
+        using var retryResponse = await _http.SendAsync(retry, token).ConfigureAwait(false);
+        retryResponse.EnsureSuccessStatusCode();
+    }
+
+    public void Dispose() => _http.Dispose();
+}