feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542)

Reshapes IP banning around one idea: **core owns the question, content owns every answer.**

Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter.

## The seam

```csharp
public interface IConnectionFilter
{
    string Name { get; }
    void Configure();
    void Start(CancellationToken token);
    void Stop();
    bool ShouldDeny(IPAddress address);
}
```

The accept path went from hardcoded branches to one question:

```csharp
else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy))
{
    logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy);
}
```

Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS.

Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's.

A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either.

This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic.

## What ships behind it

**`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands.

**`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall.

The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up.

The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves.

**`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream.

## CrowdSec: contribute-first

`IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`.

Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work.

### Why not pull decisions from CrowdSec?

The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce.

The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline.

## Threading policy

`CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem:

- Anything touching game state runs **only** on the main loop.
- Heavy work that *needs* game state must be **chunked** across ticks, never threaded.
- Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**.

Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it.

## Shared primitives

`SortedRangeIndex<T> where T : IBinaryInteger<T>` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it.

Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests.

`IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated.

## Config

| File | Owner | Keys |
|---|---|---|
| `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` |
| `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` |
| `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` |
| `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) |

Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change.

## Notes for review

- **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex<UInt128>`, same semantics and public API.
- **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`.
- **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core.
- **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel.
- `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR.
- The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message.

## Tests

**1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader.
This commit is contained in:
Kamron Batman 2026-07-25 11:59:37 -07:00 • committed by GitHub
parent bec4cfa910
commit c39454137e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
52 changed files with 4655 additions and 547 deletions

View file

@ -1,3 +1,4 @@
using System;
using System.IO;
using System.Reflection;
using System.Threading;
@ -78,6 +79,15 @@ internal static class TestServerInitializer
Core.LoopContext = new EventLoopContext();
Core.Expansion = Expansion.EJ;
// Seed the loop clock as Main.cs does before the Configure sweep; otherwise Core.Now is
// DateTime.MinValue for the whole test host.
Core._now = DateTime.UtcNow;
// Timer wheel must exist before NetState.Configure(), which schedules a recurring
// sweep via Timer.DelayCall (matches production ordering in Main.cs: Timer.Init runs
// before AssemblyHandler.Invoke("Configure")).
Timer.Init(0);
// Configure networking (initializes RingSocketManager for tests)
Server.Network.NetState.Configure();
@ -87,8 +97,6 @@ internal static class TestServerInitializer
// Configure the world
World.Configure();
Timer.Init(0);
// Load the world
World.Load();

View file

@ -0,0 +1,73 @@
using System;
using System.Collections.Generic;
using System.Net;
using System.Threading;
using Server.Network.Bans;
using Xunit;
namespace Server.Tests.Network.Bans;
public class BanChannelTests
{
private sealed class FakeReporter : IBanReporter
{
public readonly List<(IPAddress ip, TimeSpan ttl, string reason)> Reports = [];
public readonly List<IPAddress> Retractions = [];
public bool ThrowOnReport;
public string Name => "fake";
public bool CanRetract => true;
public void Register() { }
public void Start(CancellationToken token) { }
public void Stop() { }
public void Report(IPAddress address, TimeSpan ttl, string reason)
{
if (ThrowOnReport)
{
throw new InvalidOperationException("boom");
}
Reports.Add((address, ttl, reason));
}
public void Retract(IPAddress address) => Retractions.Add(address);
}
[Fact]
public void Report_FansOutToAllReporters()
{
var a = new FakeReporter();
var b = new FakeReporter();
BanChannel.ConfigureForTesting([a, b]);
BanChannel.Report(IPAddress.Parse("1.2.3.4"), TimeSpan.FromHours(1), "rate-limit");
Assert.Single(a.Reports);
Assert.Single(b.Reports);
Assert.Equal("rate-limit", a.Reports[0].reason);
}
[Fact]
public void Report_SwallowsReporterException()
{
var bad = new FakeReporter { ThrowOnReport = true };
var good = new FakeReporter();
BanChannel.ConfigureForTesting([bad, good]);
BanChannel.Report(IPAddress.Parse("1.2.3.4"), TimeSpan.FromHours(1), "manual");
Assert.Single(good.Reports); // the throwing reporter does not block the others
}
[Fact]
public void Retract_ReachesRetractCapableReporters()
{
var a = new FakeReporter();
BanChannel.ConfigureForTesting([a]);
BanChannel.Retract(IPAddress.Parse("9.9.9.9"));
Assert.Single(a.Retractions);
}
}

View file

@ -0,0 +1,44 @@
using System;
using System.Text.Json;
using Server.Json;
using Server.Network.Bans;
using Xunit;
namespace Server.Tests;
public class BanConfigurationTests
{
// Locks the JsonConfig casing/converter contract: JsonConfig's options are case-SENSITIVE, so
// every settings member must carry an explicit [JsonPropertyName("camelCase")] or it silently
// binds nothing. These tests round-trip through the exact options the loader uses.
[Fact]
public void BanSettings_RoundTripsThroughJsonConfig()
{
var original = new BanSettings
{
ReportRateLimitTrips = false,
AutoBanDuration = TimeSpan.FromHours(2)
};
var json = JsonConfig.Serialize(original);
Assert.Contains("\"reportRateLimitTrips\"", json);
Assert.Contains("\"autoBanDuration\"", json);
var restored = JsonSerializer.Deserialize<BanSettings>(json, JsonConfig.DefaultOptions);
Assert.NotNull(restored);
Assert.Equal(original.ReportRateLimitTrips, restored.ReportRateLimitTrips);
Assert.Equal(original.AutoBanDuration, restored.AutoBanDuration); // TimeSpan survives
}
[Fact]
public void BanSettings_Defaults_AreReportRateLimitTripsFourHourAutoBan()
{
var settings = new BanSettings();
Assert.True(settings.ReportRateLimitTrips);
Assert.Equal(TimeSpan.FromHours(4), settings.AutoBanDuration);
}
}

View file

@ -0,0 +1,133 @@
/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: ConnectionFiltersTests.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Net;
using System.Threading;
using Server.Network;
using Xunit;
namespace Server.Tests.Network;
[Collection("Sequential Server Tests")]
public class ConnectionFiltersTests : IDisposable
{
public ConnectionFiltersTests() => ConnectionFilters.ResetForTesting();
public void Dispose() => ConnectionFilters.ResetForTesting();
[Fact]
public void No_filters_denies_nothing()
{
Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out var deniedBy));
Assert.Null(deniedBy);
}
[Fact]
public void Register_is_idempotent_by_name()
{
ConnectionFilters.Register(new FakeFilter("dupe", deny: false));
ConnectionFilters.Register(new FakeFilter("dupe", deny: true));
// The second registration is ignored, so the deny:true instance never gets consulted.
Assert.Single(ConnectionFilters.Filters);
Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out _));
}
[Fact]
public void First_denying_filter_short_circuits_and_is_named()
{
var first = new FakeFilter("allow-all", deny: false);
var second = new FakeFilter("deny-all", deny: true);
var third = new FakeFilter("never-reached", deny: true);
ConnectionFilters.Register(first);
ConnectionFilters.Register(second);
ConnectionFilters.Register(third);
Assert.True(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out var deniedBy));
Assert.Equal("deny-all", deniedBy);
Assert.Equal(1, first.Calls);
Assert.Equal(1, second.Calls);
Assert.Equal(0, third.Calls); // short-circuited
}
// A filter that throws once throws for every subsequent connection, which would turn one bug into an
// exception per accept. It must be dropped, and the connection must fail open rather than be denied
// by a filter that never actually answered.
[Fact]
public void Throwing_filter_is_unregistered_and_fails_open()
{
var bad = new FakeFilter("bad", deny: true, throws: true);
var good = new FakeFilter("good", deny: false);
ConnectionFilters.Register(bad);
ConnectionFilters.Register(good);
Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out _));
Assert.Single(ConnectionFilters.Filters);
Assert.Equal("good", ConnectionFilters.Filters[0].Name);
// Remaining filters still run on the same pass the faulty one was dropped in.
Assert.Equal(1, good.Calls);
}
[Fact]
public void Register_configures_immediately()
{
var filter = new FakeFilter("cfg", deny: false);
ConnectionFilters.Register(filter);
Assert.True(filter.Configured);
}
private sealed class FakeFilter : IConnectionFilter
{
private readonly bool _deny;
private readonly bool _throws;
public FakeFilter(string name, bool deny, bool throws = false)
{
Name = name;
_deny = deny;
_throws = throws;
}
public string Name { get; }
public int Calls { get; private set; }
public bool Configured { get; private set; }
public void Register() => Configured = true;
public void Start(CancellationToken token)
{
}
public void Stop()
{
}
public bool ShouldDeny(IPAddress address)
{
Calls++;
if (_throws)
{
throw new InvalidOperationException("simulated filter bug");
}
return _deny;
}
}
}

View file

@ -1,35 +0,0 @@
using System.Net;
using Server.Network;
using Xunit;
namespace Server.Tests;
public class FirewallEntryTests
{
[Theory]
[InlineData("192.168.1.1", "192.168.1.1")]
[InlineData("::ffff:192.168.1.1", "192.168.1.1")]
[InlineData("ae45:c5c7:9372:2d3a:413c:6490:017d:2c18", "ae45:c5c7:9372:2d3a:413c:6490:017d:2c18")]
public void TestSingleIpFirewallEntry(string ip, string startAndEndIp)
{
var entry = new SingleIpFirewallEntry(ip);
Assert.Equal(entry.MaxIpAddress, entry.MinIpAddress);
Assert.Equal(IPAddress.Parse(startAndEndIp), entry.MinIpAddress.ToIpAddress());
}
[Theory]
[InlineData("192.168.1.1/24", "192.168.1.0", "192.168.1.255")]
[InlineData("::ffff:10.25.3.250/112", "10.25.0.0", "10.25.255.255")]
[InlineData("::ffff:10.25.5.250/124", "10.25.5.240", "10.25.5.255")]
[InlineData("::ffff:192.168.1.1/120", "192.168.1.0", "192.168.1.255")]
[InlineData("d15e:d490:03cd:f9e1:95d8:8413:e6b8:e226/88", "D15E:D490:03CD:F9E1:95D8:8400::", "D15E:D490:03CD:F9E1:95D8:84FF:FFFF:FFFF")]
[InlineData("2001:4860:4860::8888/32", "2001:4860:0000:0000:0000:0000:0000:0000", "2001:4860:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF")]
public void TestCidrPatternIpFirewallEntry(string cidr, string startIp, string endIp)
{
var entry = new CidrFirewallEntry(cidr);
Assert.Equal(IPAddress.Parse(startIp), entry.MinIpAddress.ToIpAddress());
Assert.Equal(IPAddress.Parse(endIp), entry.MaxIpAddress.ToIpAddress());
}
}

View file

@ -1,137 +0,0 @@
using System.Net;
using System.Threading.Tasks;
using Server.Network;
using Xunit;
namespace Server.Tests;
public class FirewallTests
{
[Fact]
public void Firewall_BlocksIPAddress_WhenAdded()
{
var ip = IPAddress.Parse("192.168.1.1");
var entry = new SingleIpFirewallEntry("192.168.1.1");
Assert.False(Firewall.IsBlocked(ip));
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(ip));
}
[Fact]
public void Firewall_DoesNotBlockIPAddress_WhenNotAdded()
{
var ip = IPAddress.Parse("192.168.1.2");
Assert.False(Firewall.IsBlocked(ip));
}
[Fact]
public void Firewall_StopsBlockingIPAddress_WhenRemoved()
{
var ip = IPAddress.Parse("192.168.1.3");
var entry = new SingleIpFirewallEntry("192.168.1.3");
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(ip));
Firewall.Remove(entry);
Assert.False(Firewall.IsBlocked(ip));
}
[Fact]
public void Firewall_BlocksIPRange()
{
var entry = new CidrFirewallEntry(IPAddress.Parse("10.0.0.1"), IPAddress.Parse("10.0.0.5"));
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.1")));
Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.3")));
Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.5")));
Assert.False(Firewall.IsBlocked(IPAddress.Parse("10.0.0.6")));
}
[Fact]
public void Firewall_CacheInvalidation_WorksOnUpdate()
{
var ip = IPAddress.Parse("192.168.1.10");
var entry = new SingleIpFirewallEntry("192.168.1.10");
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(ip));
Firewall.Remove(entry);
Assert.False(Firewall.IsBlocked(ip));
}
[Fact]
public void Firewall_ReadsFirewallSetCorrectly()
{
var entry = new SingleIpFirewallEntry("172.16.0.1");
Firewall.Add(entry);
var found = false;
Firewall.ReadFirewallSet(set =>
{
found = set.Contains(entry);
});
Assert.True(found);
}
[Fact]
public void Firewall_IsThreadSafe()
{
var testIps = new IPAddress[256];
for (var i = 0; i <= 255; i++)
{
testIps[i] = IPAddress.Parse($"192.168.0.{i}");
}
var entry = new CidrFirewallEntry(IPAddress.Parse("192.168.0.1"), IPAddress.Parse("192.168.0.255"));
Firewall.Add(entry);
Parallel.ForEach(testIps, ip =>
{
var shouldBlock = int.Parse(ip.ToString().Split('.')[3]) is > 0;
Assert.Equal(shouldBlock, Firewall.IsBlocked(ip));
});
Firewall.Remove(entry);
Parallel.ForEach(testIps, ip =>
{
Assert.False(Firewall.IsBlocked(ip));
});
}
[Fact]
public void Firewall_DoesNotThrowWhenRemovingNonExistentEntry()
{
var entry = new SingleIpFirewallEntry("203.0.113.5");
Assert.False(Firewall.Remove(entry));
}
[Fact]
public void Firewall_CacheHandlesMultipleUpdates()
{
var ip = IPAddress.Parse("192.168.1.20");
var entry = new SingleIpFirewallEntry("192.168.1.20");
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(ip));
Firewall.Remove(entry);
Assert.False(Firewall.IsBlocked(ip));
Firewall.Add(entry);
Assert.True(Firewall.IsBlocked(ip));
Firewall.Remove(entry);
Assert.False(Firewall.IsBlocked(ip));
}
}