From c39454137e6f99491f5f98082a7cbeeb7c3a7be3 Mon Sep 17 00:00:00 2001 From: Kamron Batman <3953314+kamronbatman@users.noreply.github.com> Date: Sat, 25 Jul 2026 11:59:37 -0700 Subject: [PATCH] feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reshapes IP banning around one idea: **core owns the question, content owns every answer.** Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter. ## The seam ```csharp public interface IConnectionFilter { string Name { get; } void Configure(); void Start(CancellationToken token); void Stop(); bool ShouldDeny(IPAddress address); } ``` The accept path went from hardcoded branches to one question: ```csharp else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy)) { logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy); } ``` Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS. Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's. A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either. This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic. ## What ships behind it **`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands. **`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall. The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up. The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves. **`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream. ## CrowdSec: contribute-first `IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`. Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work. ### Why not pull decisions from CrowdSec? The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce. The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline. ## Threading policy `CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem: - Anything touching game state runs **only** on the main loop. - Heavy work that *needs* game state must be **chunked** across ticks, never threaded. - Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**. Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it. ## Shared primitives `SortedRangeIndex where T : IBinaryInteger` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it. Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests. `IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated. ## Config | File | Owner | Keys | |---|---|---| | `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` | | `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` | | `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` | | `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) | Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change. ## Notes for review - **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex`, same semantics and public API. - **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`. - **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core. - **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel. - `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR. - The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message. ## Tests **1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader. --- .gitignore | 12 +- CLAUDE.md | 4 +- .../Fixtures/TestServerInitializer.cs | 12 +- .../Tests/Network/Bans/BanChannelTests.cs | 73 +++ .../Network/Bans/BanConfigurationTests.cs | 44 ++ .../Tests/Network/ConnectionFiltersTests.cs | 133 +++++ .../Tests/Network/Firewall/FirewallTests.cs | 137 ----- .../Server/Collections/SortedRangeIndex.cs | 134 +++++ Projects/Server/Main.cs | 9 +- Projects/Server/Network/Bans/BanChannel.cs | 143 +++++ .../Server/Network/Bans/BanConfiguration.cs | 76 +++ Projects/Server/Network/Bans/IBanReporter.cs | 55 ++ Projects/Server/Network/ConnectionFilters.cs | 155 +++++ Projects/Server/Network/Firewall/Firewall.cs | 168 ------ Projects/Server/Network/IConnectionFilter.cs | 60 ++ .../Network/NetState/NetState.Network.cs | 13 +- Projects/Server/Utilities/IPAddressUtility.cs | 240 ++++++++ Projects/Server/Utilities/NetworkUtilities.cs | 53 +- Projects/Server/Utilities/Utility.cs | 39 -- .../Fixtures/TestServerInitializer.cs | 12 +- .../Blocklist/BlocklistConfigurationTests.cs | 65 +++ .../Bans/Blocklist/BlocklistFileTests.cs | 85 +++ .../Bans/Blocklist/BlocklistFilterTests.cs | 94 +++ .../Bans/Blocklist/BlocklistSnapshotTests.cs | 98 ++++ .../Bans/Blocklist/PromotedGuardTests.cs | 46 ++ .../Network/Bans/CrowdSecAlertClientTests.cs | 51 ++ .../Bans/CrowdSecConfigurationTests.cs | 82 +++ .../Network/Bans/CrowdSecReporterTests.cs | 229 ++++++++ .../Network/Firewall/FirewallEntryTests.cs | 0 .../Firewall/FirewallPersistenceTests.cs | 72 +++ .../Tests/Network/Firewall/FirewallTests.cs | 89 +++ .../Commands/Generic/Commands/Commands.cs | 4 +- Projects/UOContent/Gumps/AdminGump.cs | 34 +- Projects/UOContent/Misc/AdminFirewall.cs | 139 ----- .../Misc/Blocklist/BlocklistConfiguration.cs | 90 +++ .../UOContent/Misc/Blocklist/BlocklistFile.cs | 86 +++ .../Misc/Blocklist/BlocklistFilter.cs | 259 +++++++++ .../Misc/Blocklist/BlocklistSnapshot.cs | 205 +++++++ .../UOContent/Misc/Blocklist/PromotedGuard.cs | 55 ++ .../UOContent/Misc/CrowdSec/CrowdSecAlert.cs | 65 +++ .../Misc/CrowdSec/CrowdSecAlertClient.cs | 137 +++++ .../Misc/CrowdSec/CrowdSecConfiguration.cs | 101 ++++ .../Misc/CrowdSec/CrowdSecReporter.cs | 399 +++++++++++++ .../Misc}/Firewall/BaseFirewallEntry.cs | 0 .../Misc}/Firewall/CidrFirewallEntry.cs | 31 +- Projects/UOContent/Misc/Firewall/Firewall.cs | 413 +++++++++++++ .../Misc/Firewall/FirewallConnectionFilter.cs | 52 ++ .../Misc/Firewall/FirewallSettings.cs | 42 ++ .../Misc}/Firewall/IFirewallEntry.cs | 0 .../Misc}/Firewall/SingleIpFirewallEntry.cs | 0 dev-docs/networking-packets.md | 64 +++ tools/Export-IpBlocklist.ps1 | 543 ++++++++++++++++++ 52 files changed, 4655 insertions(+), 547 deletions(-) create mode 100644 Projects/Server.Tests/Tests/Network/Bans/BanChannelTests.cs create mode 100644 Projects/Server.Tests/Tests/Network/Bans/BanConfigurationTests.cs create mode 100644 Projects/Server.Tests/Tests/Network/ConnectionFiltersTests.cs delete mode 100644 Projects/Server.Tests/Tests/Network/Firewall/FirewallTests.cs create mode 100644 Projects/Server/Collections/SortedRangeIndex.cs create mode 100644 Projects/Server/Network/Bans/BanChannel.cs create mode 100644 Projects/Server/Network/Bans/BanConfiguration.cs create mode 100644 Projects/Server/Network/Bans/IBanReporter.cs create mode 100644 Projects/Server/Network/ConnectionFilters.cs delete mode 100644 Projects/Server/Network/Firewall/Firewall.cs create mode 100644 Projects/Server/Network/IConnectionFilter.cs create mode 100644 Projects/Server/Utilities/IPAddressUtility.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistConfigurationTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFileTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFilterTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistSnapshotTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/PromotedGuardTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecAlertClientTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecConfigurationTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecReporterTests.cs rename Projects/{Server.Tests => UOContent.Tests}/Tests/Network/Firewall/FirewallEntryTests.cs (100%) create mode 100644 Projects/UOContent.Tests/Tests/Network/Firewall/FirewallPersistenceTests.cs create mode 100644 Projects/UOContent.Tests/Tests/Network/Firewall/FirewallTests.cs delete mode 100644 Projects/UOContent/Misc/AdminFirewall.cs create mode 100644 Projects/UOContent/Misc/Blocklist/BlocklistConfiguration.cs create mode 100644 Projects/UOContent/Misc/Blocklist/BlocklistFile.cs create mode 100644 Projects/UOContent/Misc/Blocklist/BlocklistFilter.cs create mode 100644 Projects/UOContent/Misc/Blocklist/BlocklistSnapshot.cs create mode 100644 Projects/UOContent/Misc/Blocklist/PromotedGuard.cs create mode 100644 Projects/UOContent/Misc/CrowdSec/CrowdSecAlert.cs create mode 100644 Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs create mode 100644 Projects/UOContent/Misc/CrowdSec/CrowdSecConfiguration.cs create mode 100644 Projects/UOContent/Misc/CrowdSec/CrowdSecReporter.cs rename Projects/{Server/Network => UOContent/Misc}/Firewall/BaseFirewallEntry.cs (100%) rename Projects/{Server/Network => UOContent/Misc}/Firewall/CidrFirewallEntry.cs (70%) create mode 100644 Projects/UOContent/Misc/Firewall/Firewall.cs create mode 100644 Projects/UOContent/Misc/Firewall/FirewallConnectionFilter.cs create mode 100644 Projects/UOContent/Misc/Firewall/FirewallSettings.cs rename Projects/{Server/Network => UOContent/Misc}/Firewall/IFirewallEntry.cs (100%) rename Projects/{Server/Network => UOContent/Misc}/Firewall/SingleIpFirewallEntry.cs (100%) create mode 100644 tools/Export-IpBlocklist.ps1 diff --git a/.gitignore b/.gitignore index 7bb942e52..222cdabf2 100644 --- a/.gitignore +++ b/.gitignore @@ -9,7 +9,12 @@ /Distribution/bsdtar /Distribution/Configuration/antimacro.json /Distribution/Configuration/assistants.json +/Distribution/Configuration/bans.json +/Distribution/Configuration/blocklist.json +/Distribution/Configuration/crowdsec.json /Distribution/Configuration/expansion.json +/Distribution/Configuration/ip-blocklist.txt +/Distribution/Configuration/ip-blocklist.txt.tmp /Distribution/Configuration/modernuo.json /Distribution/Configuration/email-settings.json /Distribution/Configuration/throttles.json @@ -19,6 +24,7 @@ /Distribution/Backups /Distribution/Saves /Distribution/docs +/docs/ /Distribution/temp /Distribution/*.dylib /Distribution/*.so @@ -46,5 +52,7 @@ /packages/* /Distribution/Configuration/server-access.json -# BuildTool native binaries (downloaded from GitHub Releases) -/tools/ +# BuildTool native binaries (downloaded from GitHub Releases). +# Ignore everything under tools/ except the operator scripts checked in below. +/tools/* +!/tools/Export-IpBlocklist.ps1 diff --git a/CLAUDE.md b/CLAUDE.md index 5335c036d..1b52b6816 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,14 +12,14 @@ Apply these when writing or reviewing `.cs` files under `Projects/`. 1. **LINQ** — Tier 1 (zero-cost patterns) free on hot paths; Tier 2 (low overhead) OK on warm paths; Tier 3 (allocating) forbidden on hot paths → `dev-docs/code-standards.md` 2. **No `Console.WriteLine`** — use `LogFactory.GetLogger(typeof(MyClass))` → `logger.Information(...)` (requires `using Server.Logging;`) -3. **No concurrency primitives** — no `lock`, `volatile`, `ConcurrentDictionary`, `Mutex`, etc. Server is single-threaded. +3. **Threading policy** — game logic runs only on the main loop; **never** touch game state (`World`, mobiles, items, maps, timers) from a background thread. Heavy work that *needs* game state must be **chunked** across ticks, not threaded. Heavy work that does *not* need game state (large-file parse, external I/O) **must** run on a background thread **and must yield to world saves** (defer while `World.Saving`/`WorldState.PendingSave`). Publish results back to the loop as an immutable snapshot swapped via a single `volatile` reference — the only sanctioned `volatile`. No `lock`/`Mutex`/`ConcurrentDictionary` in game logic. Rule #10 covers how background work hands results back to the loop → `dev-docs/threading-model.md` 4. **No `World.Mobiles`/`World.Items` iteration** — use spatial queries: `map.GetMobilesInRange()`, `map.GetItemsInRange()` 5. **Clean up refs in `OnDelete()`/`OnAfterDelete()`** — null out `Item`/`Mobile` references 6. **Cancel timers in `OnDelete()`/`OnAfterDelete()`** — call `_token.Cancel()` or `_timer?.Stop()` 7. **`STArrayPool.Shared`** not `ArrayPool.Shared` — single-threaded optimized, no locks 8. **`PooledRefList`** not `new List()` on hot paths — zero GC pressure, stack-allocated ref struct 9. **Serialization** — class must be `partial`, constructor needs `[Constructible]`, `TimerExecutionToken` must NOT have `[SerializableField]`. New classes: use `[SerializationGenerator(version)]` (omit `encoded`). When bumping versions, add `MigrateFrom(VXContent)` (X = previous version). Never modify `Deserialize(reader, version)` for version bumps — that method is only for pre-codegen legacy saves. When migrating from pre-codegen Serialize/Deserialize: pass `false` if old code used `reader.ReadInt()`, bump version +1, and keep old logic as `private void Deserialize(IGenericReader reader, int version)` → `dev-docs/runuo-migration-docs/02-serialization.md` -10. **No `Task.Run`/`new Thread()`** in game code — game logic is single-threaded event loop +10. **No `Task.Run`/`new Thread()` for game logic** (tandem with rule #3) — game logic is the single-threaded event loop. Backgrounding is allowed only for work that does not itself touch game state (external service calls, large-file parse). When such work must *feed* game logic: run the heavy/I/O part off-loop and `ConfigureAwait(false)` its awaits so a continuation never resumes on the loop and silently foregrounds heavy work; then hand the result back **explicitly** — publish an immutable snapshot swapped via a `volatile` reference (the loop reads it lock-free), or marshal the apply step with `Core.LoopContext.Post(() => …)`. Never touch game state off-thread; never let the scheduler decide where the heavy work runs → `dev-docs/threading-model.md` 11. **Never assume era** — if code uses `Core.AOS`/`Core.SE`/etc., ask which expansion to target 12. **Naming** — `_camelCase` private fields, `PascalCase` properties/methods/classes; don't flag legacy `m_` but use `_` for new code 13. **No empty gumps** — every gump must produce visual elements. An empty gump leaks on client+server (no way to close it). Use static `DisplayTo()` to validate before constructing → `dev-docs/gump-system.md` diff --git a/Projects/Server.Tests/Fixtures/TestServerInitializer.cs b/Projects/Server.Tests/Fixtures/TestServerInitializer.cs index e84c16509..845efb115 100644 --- a/Projects/Server.Tests/Fixtures/TestServerInitializer.cs +++ b/Projects/Server.Tests/Fixtures/TestServerInitializer.cs @@ -1,3 +1,4 @@ +using System; using System.IO; using System.Reflection; using System.Threading; @@ -78,6 +79,15 @@ internal static class TestServerInitializer Core.LoopContext = new EventLoopContext(); Core.Expansion = Expansion.EJ; + // Seed the loop clock as Main.cs does before the Configure sweep; otherwise Core.Now is + // DateTime.MinValue for the whole test host. + Core._now = DateTime.UtcNow; + + // Timer wheel must exist before NetState.Configure(), which schedules a recurring + // sweep via Timer.DelayCall (matches production ordering in Main.cs: Timer.Init runs + // before AssemblyHandler.Invoke("Configure")). + Timer.Init(0); + // Configure networking (initializes RingSocketManager for tests) Server.Network.NetState.Configure(); @@ -87,8 +97,6 @@ internal static class TestServerInitializer // Configure the world World.Configure(); - Timer.Init(0); - // Load the world World.Load(); diff --git a/Projects/Server.Tests/Tests/Network/Bans/BanChannelTests.cs b/Projects/Server.Tests/Tests/Network/Bans/BanChannelTests.cs new file mode 100644 index 000000000..a48647152 --- /dev/null +++ b/Projects/Server.Tests/Tests/Network/Bans/BanChannelTests.cs @@ -0,0 +1,73 @@ +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans; + +public class BanChannelTests +{ + private sealed class FakeReporter : IBanReporter + { + public readonly List<(IPAddress ip, TimeSpan ttl, string reason)> Reports = []; + public readonly List Retractions = []; + public bool ThrowOnReport; + + public string Name => "fake"; + public bool CanRetract => true; + public void Register() { } + public void Start(CancellationToken token) { } + public void Stop() { } + + public void Report(IPAddress address, TimeSpan ttl, string reason) + { + if (ThrowOnReport) + { + throw new InvalidOperationException("boom"); + } + + Reports.Add((address, ttl, reason)); + } + + public void Retract(IPAddress address) => Retractions.Add(address); + } + + [Fact] + public void Report_FansOutToAllReporters() + { + var a = new FakeReporter(); + var b = new FakeReporter(); + BanChannel.ConfigureForTesting([a, b]); + + BanChannel.Report(IPAddress.Parse("1.2.3.4"), TimeSpan.FromHours(1), "rate-limit"); + + Assert.Single(a.Reports); + Assert.Single(b.Reports); + Assert.Equal("rate-limit", a.Reports[0].reason); + } + + [Fact] + public void Report_SwallowsReporterException() + { + var bad = new FakeReporter { ThrowOnReport = true }; + var good = new FakeReporter(); + BanChannel.ConfigureForTesting([bad, good]); + + BanChannel.Report(IPAddress.Parse("1.2.3.4"), TimeSpan.FromHours(1), "manual"); + + Assert.Single(good.Reports); // the throwing reporter does not block the others + } + + [Fact] + public void Retract_ReachesRetractCapableReporters() + { + var a = new FakeReporter(); + BanChannel.ConfigureForTesting([a]); + + BanChannel.Retract(IPAddress.Parse("9.9.9.9")); + + Assert.Single(a.Retractions); + } +} diff --git a/Projects/Server.Tests/Tests/Network/Bans/BanConfigurationTests.cs b/Projects/Server.Tests/Tests/Network/Bans/BanConfigurationTests.cs new file mode 100644 index 000000000..ee232b1df --- /dev/null +++ b/Projects/Server.Tests/Tests/Network/Bans/BanConfigurationTests.cs @@ -0,0 +1,44 @@ +using System; +using System.Text.Json; +using Server.Json; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests; + +public class BanConfigurationTests +{ + // Locks the JsonConfig casing/converter contract: JsonConfig's options are case-SENSITIVE, so + // every settings member must carry an explicit [JsonPropertyName("camelCase")] or it silently + // binds nothing. These tests round-trip through the exact options the loader uses. + + [Fact] + public void BanSettings_RoundTripsThroughJsonConfig() + { + var original = new BanSettings + { + ReportRateLimitTrips = false, + AutoBanDuration = TimeSpan.FromHours(2) + }; + + var json = JsonConfig.Serialize(original); + + Assert.Contains("\"reportRateLimitTrips\"", json); + Assert.Contains("\"autoBanDuration\"", json); + + var restored = JsonSerializer.Deserialize(json, JsonConfig.DefaultOptions); + + Assert.NotNull(restored); + Assert.Equal(original.ReportRateLimitTrips, restored.ReportRateLimitTrips); + Assert.Equal(original.AutoBanDuration, restored.AutoBanDuration); // TimeSpan survives + } + + [Fact] + public void BanSettings_Defaults_AreReportRateLimitTripsFourHourAutoBan() + { + var settings = new BanSettings(); + + Assert.True(settings.ReportRateLimitTrips); + Assert.Equal(TimeSpan.FromHours(4), settings.AutoBanDuration); + } +} diff --git a/Projects/Server.Tests/Tests/Network/ConnectionFiltersTests.cs b/Projects/Server.Tests/Tests/Network/ConnectionFiltersTests.cs new file mode 100644 index 000000000..b7e134794 --- /dev/null +++ b/Projects/Server.Tests/Tests/Network/ConnectionFiltersTests.cs @@ -0,0 +1,133 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: ConnectionFiltersTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Net; +using System.Threading; +using Server.Network; +using Xunit; + +namespace Server.Tests.Network; + +[Collection("Sequential Server Tests")] +public class ConnectionFiltersTests : IDisposable +{ + public ConnectionFiltersTests() => ConnectionFilters.ResetForTesting(); + + public void Dispose() => ConnectionFilters.ResetForTesting(); + + [Fact] + public void No_filters_denies_nothing() + { + Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out var deniedBy)); + Assert.Null(deniedBy); + } + + [Fact] + public void Register_is_idempotent_by_name() + { + ConnectionFilters.Register(new FakeFilter("dupe", deny: false)); + ConnectionFilters.Register(new FakeFilter("dupe", deny: true)); + + // The second registration is ignored, so the deny:true instance never gets consulted. + Assert.Single(ConnectionFilters.Filters); + Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out _)); + } + + [Fact] + public void First_denying_filter_short_circuits_and_is_named() + { + var first = new FakeFilter("allow-all", deny: false); + var second = new FakeFilter("deny-all", deny: true); + var third = new FakeFilter("never-reached", deny: true); + + ConnectionFilters.Register(first); + ConnectionFilters.Register(second); + ConnectionFilters.Register(third); + + Assert.True(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out var deniedBy)); + Assert.Equal("deny-all", deniedBy); + Assert.Equal(1, first.Calls); + Assert.Equal(1, second.Calls); + Assert.Equal(0, third.Calls); // short-circuited + } + + // A filter that throws once throws for every subsequent connection, which would turn one bug into an + // exception per accept. It must be dropped, and the connection must fail open rather than be denied + // by a filter that never actually answered. + [Fact] + public void Throwing_filter_is_unregistered_and_fails_open() + { + var bad = new FakeFilter("bad", deny: true, throws: true); + var good = new FakeFilter("good", deny: false); + + ConnectionFilters.Register(bad); + ConnectionFilters.Register(good); + + Assert.False(ConnectionFilters.ShouldDeny(IPAddress.Parse("1.2.3.4"), out _)); + Assert.Single(ConnectionFilters.Filters); + Assert.Equal("good", ConnectionFilters.Filters[0].Name); + + // Remaining filters still run on the same pass the faulty one was dropped in. + Assert.Equal(1, good.Calls); + } + + [Fact] + public void Register_configures_immediately() + { + var filter = new FakeFilter("cfg", deny: false); + ConnectionFilters.Register(filter); + + Assert.True(filter.Configured); + } + + private sealed class FakeFilter : IConnectionFilter + { + private readonly bool _deny; + private readonly bool _throws; + + public FakeFilter(string name, bool deny, bool throws = false) + { + Name = name; + _deny = deny; + _throws = throws; + } + + public string Name { get; } + public int Calls { get; private set; } + public bool Configured { get; private set; } + + public void Register() => Configured = true; + + public void Start(CancellationToken token) + { + } + + public void Stop() + { + } + + public bool ShouldDeny(IPAddress address) + { + Calls++; + if (_throws) + { + throw new InvalidOperationException("simulated filter bug"); + } + + return _deny; + } + } +} diff --git a/Projects/Server.Tests/Tests/Network/Firewall/FirewallTests.cs b/Projects/Server.Tests/Tests/Network/Firewall/FirewallTests.cs deleted file mode 100644 index f803ee586..000000000 --- a/Projects/Server.Tests/Tests/Network/Firewall/FirewallTests.cs +++ /dev/null @@ -1,137 +0,0 @@ - -using System.Net; -using System.Threading.Tasks; -using Server.Network; -using Xunit; - -namespace Server.Tests; - -public class FirewallTests -{ - [Fact] - public void Firewall_BlocksIPAddress_WhenAdded() - { - var ip = IPAddress.Parse("192.168.1.1"); - var entry = new SingleIpFirewallEntry("192.168.1.1"); - - Assert.False(Firewall.IsBlocked(ip)); - - Firewall.Add(entry); - - Assert.True(Firewall.IsBlocked(ip)); - } - - [Fact] - public void Firewall_DoesNotBlockIPAddress_WhenNotAdded() - { - var ip = IPAddress.Parse("192.168.1.2"); - Assert.False(Firewall.IsBlocked(ip)); - } - - [Fact] - public void Firewall_StopsBlockingIPAddress_WhenRemoved() - { - var ip = IPAddress.Parse("192.168.1.3"); - var entry = new SingleIpFirewallEntry("192.168.1.3"); - - Firewall.Add(entry); - Assert.True(Firewall.IsBlocked(ip)); - - Firewall.Remove(entry); - Assert.False(Firewall.IsBlocked(ip)); - } - - [Fact] - public void Firewall_BlocksIPRange() - { - var entry = new CidrFirewallEntry(IPAddress.Parse("10.0.0.1"), IPAddress.Parse("10.0.0.5")); - - Firewall.Add(entry); - - Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.1"))); - Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.3"))); - Assert.True(Firewall.IsBlocked(IPAddress.Parse("10.0.0.5"))); - - Assert.False(Firewall.IsBlocked(IPAddress.Parse("10.0.0.6"))); - } - - [Fact] - public void Firewall_CacheInvalidation_WorksOnUpdate() - { - var ip = IPAddress.Parse("192.168.1.10"); - var entry = new SingleIpFirewallEntry("192.168.1.10"); - - Firewall.Add(entry); - Assert.True(Firewall.IsBlocked(ip)); - - Firewall.Remove(entry); - Assert.False(Firewall.IsBlocked(ip)); - } - - [Fact] - public void Firewall_ReadsFirewallSetCorrectly() - { - var entry = new SingleIpFirewallEntry("172.16.0.1"); - Firewall.Add(entry); - - var found = false; - Firewall.ReadFirewallSet(set => - { - found = set.Contains(entry); - }); - - Assert.True(found); - } - - [Fact] - public void Firewall_IsThreadSafe() - { - var testIps = new IPAddress[256]; - for (var i = 0; i <= 255; i++) - { - testIps[i] = IPAddress.Parse($"192.168.0.{i}"); - } - - var entry = new CidrFirewallEntry(IPAddress.Parse("192.168.0.1"), IPAddress.Parse("192.168.0.255")); - Firewall.Add(entry); - - Parallel.ForEach(testIps, ip => - { - var shouldBlock = int.Parse(ip.ToString().Split('.')[3]) is > 0; - Assert.Equal(shouldBlock, Firewall.IsBlocked(ip)); - }); - - Firewall.Remove(entry); - - Parallel.ForEach(testIps, ip => - { - Assert.False(Firewall.IsBlocked(ip)); - }); - } - - [Fact] - public void Firewall_DoesNotThrowWhenRemovingNonExistentEntry() - { - var entry = new SingleIpFirewallEntry("203.0.113.5"); - Assert.False(Firewall.Remove(entry)); - } - - [Fact] - public void Firewall_CacheHandlesMultipleUpdates() - { - var ip = IPAddress.Parse("192.168.1.20"); - var entry = new SingleIpFirewallEntry("192.168.1.20"); - - Firewall.Add(entry); - Assert.True(Firewall.IsBlocked(ip)); - - Firewall.Remove(entry); - Assert.False(Firewall.IsBlocked(ip)); - - Firewall.Add(entry); - Assert.True(Firewall.IsBlocked(ip)); - - Firewall.Remove(entry); - Assert.False(Firewall.IsBlocked(ip)); - } -} diff --git a/Projects/Server/Collections/SortedRangeIndex.cs b/Projects/Server/Collections/SortedRangeIndex.cs new file mode 100644 index 000000000..5af37fd59 --- /dev/null +++ b/Projects/Server/Collections/SortedRangeIndex.cs @@ -0,0 +1,134 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: SortedRangeIndex.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Numerics; + +namespace Server.Collections; + +/// +/// Immutable, allocation-lean membership index over a set of inclusive integer ranges. Ranges are +/// stored as two parallel arrays (_mins/_maxs) sorted by minimum and coalesced into +/// disjoint runs, so a single binary search decides membership. Coalescing is required for +/// correctness: only inspects the rightmost run whose minimum is <= the +/// value, which is only sound when the runs never overlap or nest. +/// +public sealed class SortedRangeIndex where T : IBinaryInteger +{ + public static readonly SortedRangeIndex Empty = new([], []); + + /// An inclusive [Min, Max] range. Singles are represented as Min == Max. + public readonly record struct Range(T Min, T Max); + + /// Orders ranges ascending by minimum; the coalescing pass in requires this. + public static readonly Comparison ByMin = static (a, b) => a.Min.CompareTo(b.Min); + + private readonly T[] _mins; + private readonly T[] _maxs; + + private SortedRangeIndex(T[] mins, T[] maxs) + { + _mins = mins; + _maxs = maxs; + } + + public int Count => _mins.Length; + + /// + /// True when falls in any range. Binary-searches for the rightmost run + /// whose minimum is <= the value, then tests that value against that run's maximum. + /// + public bool Contains(T value) + { + var lo = 0; + var hi = _mins.Length - 1; + var found = -1; + while (lo <= hi) + { + var mid = (lo + hi) >> 1; + if (_mins[mid] <= value) + { + found = mid; + lo = mid + 1; + } + else + { + hi = mid - 1; + } + } + + return found >= 0 && value <= _maxs[found]; + } + + /// + /// Builds an index from ranges that are already sorted ascending by (sort + /// the source with first). Overlapping and nested ranges are merged into disjoint + /// runs. Two passes over the (pooled) input keep the run count exact so only the two final arrays are + /// heap-allocated: pass one counts the runs, pass two fills the exact-size arrays. + /// + public static SortedRangeIndex Build(ReadOnlySpan sortedByMin) + { + if (sortedByMin.IsEmpty) + { + return Empty; + } + + // Pass 1: count the disjoint runs so the final arrays can be sized exactly. + var runs = 1; + var curMax = sortedByMin[0].Max; + for (var i = 1; i < sortedByMin.Length; i++) + { + var r = sortedByMin[i]; + if (r.Min <= curMax) + { + if (r.Max > curMax) + { + curMax = r.Max; + } + } + else + { + runs++; + curMax = r.Max; + } + } + + // Pass 2: write the coalesced runs into the exact-size final arrays. + var mins = new T[runs]; + var maxs = new T[runs]; + var w = 0; + mins[0] = sortedByMin[0].Min; + maxs[0] = sortedByMin[0].Max; + for (var i = 1; i < sortedByMin.Length; i++) + { + var r = sortedByMin[i]; + if (r.Min <= maxs[w]) + { + if (r.Max > maxs[w]) + { + maxs[w] = r.Max; + } + } + else + { + w++; + mins[w] = r.Min; + maxs[w] = r.Max; + } + } + + return new SortedRangeIndex(mins, maxs); + } +} diff --git a/Projects/Server/Main.cs b/Projects/Server/Main.cs index f89fd3084..2be9d7d04 100644 --- a/Projects/Server/Main.cs +++ b/Projects/Server/Main.cs @@ -1,4 +1,4 @@ -/************************************************************************* +/************************************************************************* * ModernUO * * Copyright 2019-2026 - ModernUO Development Team * * Email: hi@modernuo.com * @@ -30,6 +30,7 @@ using Server.Compression; using Server.Json; using Server.Logging; using Server.Network; +using Server.Network.Bans; using Server.Text; namespace Server; @@ -260,7 +261,7 @@ public static class Core // ignored } - if (!close && !Core.Headless) + if (!close && !Headless) { Console.WriteLine("This exception is fatal, press return to exit"); ConsoleInputHandler.ReadLine(); @@ -342,6 +343,8 @@ public static class Core World.ExitSerializationThreads(); PingServer.Shutdown(); NetState.Shutdown(); + BanChannel.Stop(); + ConnectionFilters.Stop(); if (!_crashed) { @@ -461,6 +464,8 @@ public static class Core AssemblyHandler.Invoke("Initialize"); + BanChannel.Start(ClosingTokenSource.Token); + ConnectionFilters.Start(ClosingTokenSource.Token); NetState.Start(); PingServer.Start(); EventSink.InvokeServerStarted(); diff --git a/Projects/Server/Network/Bans/BanChannel.cs b/Projects/Server/Network/Bans/BanChannel.cs new file mode 100644 index 000000000..6d4ead9b1 --- /dev/null +++ b/Projects/Server/Network/Bans/BanChannel.cs @@ -0,0 +1,143 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BanChannel.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading; +using Server.Logging; + +namespace Server.Network.Bans; + +/// +/// Coordinates the configured contribution sinks. Enforcement is NOT here — +/// the accept path asks . This channel only fans locally-decided bans out +/// to external systems (CrowdSec), which distribute them to OS-level bouncers. +/// +public static class BanChannel +{ + private static readonly ILogger logger = LogFactory.GetLogger(typeof(BanChannel)); + + private static IBanReporter[] _reporters = []; + + public static IReadOnlyList Reporters => _reporters; + + /// + /// Registers a contribution sink from content (inversion of control). Idempotent by + /// : a second registration of the same name is ignored. Configures the + /// reporter immediately so it is ready before . + /// + public static void Register(IBanReporter reporter) + { + if (reporter == null) + { + return; + } + + var reporters = _reporters; + for (var i = 0; i < reporters.Length; i++) + { + if (reporters[i].Name == reporter.Name) + { + return; + } + } + + reporter.Register(); + + var updated = new IBanReporter[_reporters.Length + 1]; + Array.Copy(_reporters, updated, _reporters.Length); + updated[^1] = reporter; + _reporters = updated; + + logger.Information("Ban channel registered reporter '{Name}'", reporter.Name); + } + + internal static void ConfigureForTesting(IBanReporter[] reporters) => _reporters = reporters ?? []; + + public static void Start(CancellationToken token) + { + var reporters = _reporters; + for (var i = 0; i < reporters.Length; i++) + { + var reporter = reporters[i]; + try + { + reporter.Start(token); + } + catch (Exception e) + { + // A broken contribution path must not crash boot — enforcement is local and unaffected. + logger.Error(e, "Ban reporter '{Name}' failed to start; continuing without it", reporter.Name); + } + } + } + + public static void Stop() + { + var reporters = _reporters; + for (var i = 0; i < reporters.Length; i++) + { + var reporter = reporters[i]; + try + { + reporter.Stop(); + } + catch (Exception e) + { + logger.Warning(e, "Ban reporter '{Name}' threw while stopping", reporter.Name); + } + } + } + + /// Fans a locally-decided ban out to every reporter. Non-blocking; never throws. + public static void Report(IPAddress ip, TimeSpan ttl, string reason) + { + var reporters = _reporters; + for (var i = 0; i < reporters.Length; i++) + { + try + { + reporters[i].Report(ip, ttl, reason); + } + catch (Exception e) + { + logger.Warning(e, "Ban reporter '{Name}' threw during Report", reporters[i].Name); + } + } + } + + /// Fans a retraction (manual unban) out to every retract-capable reporter. + public static void Retract(IPAddress ip) + { + var reporters = _reporters; + for (var i = 0; i < reporters.Length; i++) + { + if (!reporters[i].CanRetract) + { + continue; + } + + try + { + reporters[i].Retract(ip); + } + catch (Exception e) + { + logger.Warning(e, "Ban reporter '{Name}' threw during Retract", reporters[i].Name); + } + } + } +} diff --git a/Projects/Server/Network/Bans/BanConfiguration.cs b/Projects/Server/Network/Bans/BanConfiguration.cs new file mode 100644 index 000000000..2a2af40cf --- /dev/null +++ b/Projects/Server/Network/Bans/BanConfiguration.cs @@ -0,0 +1,76 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BanConfiguration.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; +using System.Text.Json.Serialization; +using Server.Json; + +namespace Server.Network.Bans; + +/// +/// Loads the from Configuration/bans.json (matching the per-feature +/// JSON config pattern used by AssistantConfiguration). Loaded once; a missing file writes a +/// local-only, fail-open template so operators have something to edit. +/// +public static class BanConfiguration +{ + private const string _path = "Configuration/bans.json"; + + public static BanSettings Settings { get; private set; } + + public static void Configure() + { + // Idempotent: a second call must not re-deserialize or overwrite an operator's edits. + if (Settings != null) + { + return; + } + + var path = Path.Join(Core.BaseDirectory, _path); + + if (File.Exists(path)) + { + Settings = JsonConfig.Deserialize(path); + } + else + { + Settings = new BanSettings + { + ReportRateLimitTrips = true, + AutoBanDuration = TimeSpan.FromHours(4) + }; + + Save(); + } + } + + private static void Save() + { + JsonConfig.Serialize(Path.Join(Core.BaseDirectory, _path), Settings); + } +} + +/// Ban-channel policy: which reporters receive contributions, and how auto-detections are handled. +public record BanSettings +{ + /// Whether IP rate-limiter trips are contributed to reporters. They never enter the local firewall set. + [JsonPropertyName("reportRateLimitTrips")] + public bool ReportRateLimitTrips { get; set; } = true; + + /// Duration reported for an auto-detected (rate-limit) ban. + [JsonPropertyName("autoBanDuration")] + public TimeSpan AutoBanDuration { get; set; } = TimeSpan.FromHours(4); +} diff --git a/Projects/Server/Network/Bans/IBanReporter.cs b/Projects/Server/Network/Bans/IBanReporter.cs new file mode 100644 index 000000000..190c6936e --- /dev/null +++ b/Projects/Server/Network/Bans/IBanReporter.cs @@ -0,0 +1,55 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: IBanReporter.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Net; +using System.Threading; + +namespace Server.Network.Bans; + +/// +/// A contribution sink behind . Reporters receive locally-decided bans +/// (manual admin bans, rate-limit trips, blocklist promotions) and forward them to an external system +/// (e.g. CrowdSec), which distributes them to OS-level bouncers. Reporters never answer the accept-path +/// membership query — that is an 's job. +/// +public interface IBanReporter +{ + /// Stable id for logging/config (e.g. crowdsec). + string Name { get; } + + /// Reads configuration. No network or file I/O here. + void Register(); + + /// Starts background delivery. The token is cancelled on shutdown. + void Start(CancellationToken token); + + /// Flushes and tears down background delivery. + void Stop(); + + /// + /// Enqueues a ban contribution. MUST be non-blocking and safe on the accept path: it may only + /// enqueue (bounded, drop-on-overflow) and never perform synchronous I/O. + /// + /// or negative = use the reporter's default duration. + /// Short slug (manual, rate-limit) used as the scenario suffix. + void Report(IPAddress address, TimeSpan ttl, string reason); + + /// True if this reporter can retract a previously-reported ban. + bool CanRetract { get; } + + /// Enqueues a retraction (e.g. a manual unban). No-op if unsupported. + void Retract(IPAddress address); +} diff --git a/Projects/Server/Network/ConnectionFilters.cs b/Projects/Server/Network/ConnectionFilters.cs new file mode 100644 index 000000000..c0bb0b2fb --- /dev/null +++ b/Projects/Server/Network/ConnectionFilters.cs @@ -0,0 +1,155 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: ConnectionFilters.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading; +using Server.Logging; + +namespace Server.Network; + +/// +/// Registry of the gates the accept path consults, and their lifecycle. +/// Filters are registered during the Configure sweep and the backing store is a plain array, so +/// is an indexed loop over a field read — no enumerator, no closure, no +/// allocation. The whole accept path runs on the game loop, so no synchronization is needed. +/// +public static class ConnectionFilters +{ + private static readonly ILogger logger = LogFactory.GetLogger(typeof(ConnectionFilters)); + + private static IConnectionFilter[] _filters = []; + + public static IReadOnlyList Filters => _filters; + + /// + /// Registers a gate (inversion of control, mirroring BanChannel.Register). Idempotent by + /// . Filters are consulted in registration order, so register + /// the cheapest and most selective first — core registers the firewall before content is swept. + /// + public static void Register(IConnectionFilter filter) + { + if (filter == null) + { + return; + } + + var filters = _filters; + for (var i = 0; i < filters.Length; i++) + { + if (filters[i].Name == filter.Name) + { + return; + } + } + + filter.Register(); + + var updated = new IConnectionFilter[_filters.Length + 1]; + Array.Copy(_filters, updated, _filters.Length); + updated[^1] = filter; + _filters = updated; + + logger.Information("Registered connection filter '{Name}'", filter.Name); + } + + /// + /// True when any filter denies the connection. Short-circuits on the first denial; + /// names it for logging. + /// + public static bool ShouldDeny(IPAddress address, out string deniedBy) + { + var filters = _filters; + for (var i = 0; i < filters.Length; i++) + { + // A faulty filter must not take down the accept loop for every connection. + try + { + if (filters[i].ShouldDeny(address)) + { + deniedBy = filters[i].Name; + return true; + } + } + catch (Exception e) + { + Disable(filters[i], e); + } + } + + deniedBy = null; + return false; + } + + /// + /// Drops a filter that threw on the accept path: one that faults once faults for every subsequent + /// connection, costing an exception and a log line per accept. Failing open is deliberate — a broken + /// filter must not be able to deny every connection either. + /// + private static void Disable(IConnectionFilter filter, Exception e) + { + logger.Error(e, "Connection filter '{Name}' threw on the accept path; unregistering it", filter.Name); + + var filters = _filters; + var updated = new List(filters.Length); + for (var i = 0; i < filters.Length; i++) + { + if (!ReferenceEquals(filters[i], filter)) + { + updated.Add(filters[i]); + } + } + + _filters = updated.ToArray(); + } + + public static void Start(CancellationToken token) + { + var filters = _filters; + for (var i = 0; i < filters.Length; i++) + { + var filter = filters[i]; + try + { + filter.Start(token); + } + catch (Exception e) + { + // A filter that cannot hydrate must not crash boot; it simply denies nothing. + logger.Error(e, "Connection filter '{Name}' failed to start; continuing without it", filter.Name); + } + } + } + + public static void Stop() + { + var filters = _filters; + for (var i = 0; i < filters.Length; i++) + { + var filter = filters[i]; + try + { + filter.Stop(); + } + catch (Exception e) + { + logger.Warning(e, "Connection filter '{Name}' threw while stopping", filter.Name); + } + } + } + + internal static void ResetForTesting() => _filters = []; +} diff --git a/Projects/Server/Network/Firewall/Firewall.cs b/Projects/Server/Network/Firewall/Firewall.cs deleted file mode 100644 index ffb778aa8..000000000 --- a/Projects/Server/Network/Firewall/Firewall.cs +++ /dev/null @@ -1,168 +0,0 @@ -/************************************************************************* - * ModernUO * - * Copyright 2019-2026 - ModernUO Development Team * - * Email: hi@modernuo.com * - * File: Firewall.cs * - * * - * This program is free software: you can redistribute it and/or modify * - * it under the terms of the GNU General Public License as published by * - * the Free Software Foundation, either version 3 of the License, or * - * (at your option) any later version. * - * * - * You should have received a copy of the GNU General Public License * - * along with this program. If not, see . * - *************************************************************************/ - -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Net; -using System.Runtime.CompilerServices; -using System.Threading; - -namespace Server.Network; - -public static class Firewall -{ - [ThreadStatic] - private static InternalValidationEntry _validationEntry; - private static readonly ConcurrentDictionary _isBlockedCache = []; - private static readonly ReaderWriterLockSlim _firewallLock = new(LockRecursionPolicy.NoRecursion); - - private static int _firewallVersion; - private static readonly SortedSet _firewallSet = []; - - public static int FirewallSetCount => _firewallSet.Count; - - public static void ReadFirewallSet(Action> callback) - { - _firewallLock.EnterReadLock(); - try - { - callback(_firewallSet); - } - finally - { - _firewallLock.ExitReadLock(); - } - } - - internal static bool IsBlocked(IPAddress address) - { - if (_isBlockedCache.TryGetValue(address, out var blockVersion) && blockVersion == _firewallVersion) - { - return true; - } - - if (_validationEntry == null) - { - _validationEntry = new InternalValidationEntry(address); - } - else - { - _validationEntry.Address = address; - } - - if (CheckBlocked(_validationEntry)) - { - _isBlockedCache[address] = _firewallVersion; - return true; - } - - return false; - } - - private static bool CheckBlocked(IFirewallEntry validationEntry) - { - if (_firewallSet.Count == 0) - { - return false; - } - - _firewallLock.EnterReadLock(); - try - { - var min = _firewallSet.Min; - if (validationEntry.CompareTo(min) < 0) - { - return false; - } - - // Get all entries that are lower than our validation entry - var view = _firewallSet.GetViewBetween(min, validationEntry); - - // Loop backward since there shouldn't be any entries where the Min address is higher than ours - foreach (var firewallEntry in view.Reverse()) - { - if (firewallEntry.IsBlocked(validationEntry.MinIpAddress)) - { - return true; - } - } - - return view.Max?.IsBlocked(validationEntry.MinIpAddress) == true; - } - finally - { - _firewallLock.ExitReadLock(); - } - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static bool Add(IFirewallEntry firewallEntry) - { - _firewallLock.EnterWriteLock(); - try - { - if (_firewallSet.Add(firewallEntry)) - { - Interlocked.Increment(ref _firewallVersion); // Update version - return true; - } - return false; - } - finally - { - _firewallLock.ExitWriteLock(); - } - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static bool Remove(IFirewallEntry entry) - { - if (entry == null) - { - return false; - } - - _firewallLock.EnterWriteLock(); - try - { - if (_firewallSet.Remove(entry)) - { - Interlocked.Increment(ref _firewallVersion); // Update version - return true; - } - return false; - } - finally - { - _firewallLock.ExitWriteLock(); - } - } - - private class InternalValidationEntry : BaseFirewallEntry - { - private UInt128 _address; - - public IPAddress Address - { - set => _address = value.ToUInt128(); - } - - public override UInt128 MinIpAddress => _address; - public override UInt128 MaxIpAddress => _address; - - public InternalValidationEntry(IPAddress ipAddress) => Address = ipAddress; - } -} diff --git a/Projects/Server/Network/IConnectionFilter.cs b/Projects/Server/Network/IConnectionFilter.cs new file mode 100644 index 000000000..48e971c85 --- /dev/null +++ b/Projects/Server/Network/IConnectionFilter.cs @@ -0,0 +1,60 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: IConnectionFilter.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Net; +using System.Threading; + +namespace Server.Network; + +/// +/// A gate consulted for every inbound connection, before the socket is configured and before any +/// per-connection allocation. Implementations decide membership only — the accept path neither knows +/// nor cares where a filter's data comes from, so a filter may be a handful of admin-curated entries, +/// a millions-strong list hydrated from a file, or a query against something else entirely. Core owns +/// the question; content owns every answer (see Firewall and BlocklistFilter in UOContent). +/// +/// +/// +/// runs on the game loop once per accepted socket, which is the path that has +/// to survive a DDoS. Implementations MUST be allocation-free and O(log n) at worst, MUST NOT perform +/// I/O, and MUST NOT block. Anything expensive (parsing, reloading, reporting to an external service) +/// belongs off the loop or behind a bounded, non-blocking enqueue. +/// +/// +/// Side effects that a hit implies (contributing to BanChannel, promoting to an OS firewall, +/// suppressing duplicate reports) are the filter's own business, not the accept path's. This is why +/// returns a bare bool: the accept path asks one question and does one thing. +/// +/// +public interface IConnectionFilter +{ + /// Stable id for logging/config (e.g. firewall, blocklist). + string Name { get; } + + /// Reads configuration. Called by . No I/O beyond config. + void Register(); + + /// Starts any background hydration. The token is cancelled on shutdown. + void Start(CancellationToken token); + + /// Flushes and tears down. Called during shutdown. + void Stop(); + + /// + /// True to deny the connection. Must be allocation-free and non-blocking; see the remarks on + /// . + /// + bool ShouldDeny(IPAddress address); +} diff --git a/Projects/Server/Network/NetState/NetState.Network.cs b/Projects/Server/Network/NetState/NetState.Network.cs index 9bfd39673..b38c88d23 100644 --- a/Projects/Server/Network/NetState/NetState.Network.cs +++ b/Projects/Server/Network/NetState/NetState.Network.cs @@ -224,10 +224,19 @@ public partial class NetState if (_ipRateLimiter != null && !_ipRateLimiter.Verify(remoteIP, out var totalAttempts)) { logger.Debug("{Address} Past IP limit threshold ({TotalAttempts})", remoteIP, totalAttempts); + + if (Bans.BanConfiguration.Settings.ReportRateLimitTrips) + { + // Enqueue-only contribution; NOT added to the local firewall set (the limiter already + // gates it here and the OS bouncer drops it at the kernel). + Bans.BanChannel.Report(remoteIP, Bans.BanConfiguration.Settings.AutoBanDuration, "rate-limit"); + } } - else if (Firewall.IsBlocked(remoteIP)) + else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy)) { - logger.Debug("{Address} Firewalled", remoteIP); + // Whatever a hit implies (persisting, promoting to an OS bouncer, contributing to the + // ban channel) is the filter's own business; the accept path just drops the socket. + logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy); } else { diff --git a/Projects/Server/Utilities/IPAddressUtility.cs b/Projects/Server/Utilities/IPAddressUtility.cs new file mode 100644 index 000000000..12f81e516 --- /dev/null +++ b/Projects/Server/Utilities/IPAddressUtility.cs @@ -0,0 +1,240 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: IPAddressUtility.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Buffers.Binary; +using System.Net; +using System.Net.Sockets; +using System.Numerics; + +namespace Server; + +/// +/// Low-level IPAddress conversion and parsing helpers shared by the firewall, ban channel, and +/// blocklist. All members are allocation-free (stack buffers only) so they are safe on hot accept +/// paths and inside tight parse loops. +/// +public static class IPAddressUtility +{ + // Converts an IPAddress to a UInt128 in IPv6 format. + // The IsIPv4MappedToIPv6 clause below looks redundant (the BCL only ever sets it on InterNetworkV6), + // but it guards the v4 -> UInt128 -> IPAddress round-trip, which can return a mapped v6 address for + // what is really a v4 one. + //TODO Rework as an explicit "to canonical v6 bits" step that needs no family check + // (see dev-docs/networking-packets.md, "IP Address Normalization") + public static UInt128 ToUInt128(this IPAddress ip) + { + if (ip.AddressFamily == AddressFamily.InterNetwork && !ip.IsIPv4MappedToIPv6) + { + Span integer = stackalloc byte[4]; + return !ip.TryWriteBytes(integer, out _) + ? (UInt128)0 + : new UInt128(0, 0xFFFF00000000UL | BinaryPrimitives.ReadUInt32BigEndian(integer)); + } + + Span bytes = stackalloc byte[16]; + if (!ip.TryWriteBytes(bytes, out _)) + { + return 0; + } + + var high = BinaryPrimitives.ReadUInt64BigEndian(bytes[..8]); + var low = BinaryPrimitives.ReadUInt64BigEndian(bytes.Slice(8, 8)); + + return new UInt128(high, low); + } + + // Converts a UInt128 in IPv6 format to an IPAddress + public static IPAddress ToIpAddress(this UInt128 value, bool mapToIpv6 = false) + { + // IPv4 mapped IPv6 address + if (!mapToIpv6 && value >= 0xFFFF00000000UL && value <= 0xFFFFFFFFFFFFUL) + { + var newAddress = IPAddress.HostToNetworkOrder((int)value); + return new IPAddress(unchecked((uint)newAddress)); + } + + Span bytes = stackalloc byte[16]; // 128 bits for IPv6 address + ((IBinaryInteger)value).WriteBigEndian(bytes); + + return new IPAddress(bytes); + } + + /// + /// Parses a.b.c.d/n, ::/n, or a bare address (treated as a single-host range) into an + /// inclusive range in normalized IPv6 form. A bare IPv4 prefix is widened by 96 + /// bits so v4 and v6 ranges are directly comparable. Returns false on anything malformed. + /// + public static bool TryParseCidrRange(ReadOnlySpan cidr, out UInt128 min, out UInt128 max) + { + min = default; + max = default; + + var slash = cidr.IndexOf('/'); + if (!IPAddress.TryParse(slash >= 0 ? cidr[..slash] : cidr, out var ip)) + { + return false; + } + + var isV6 = ip.AddressFamily == AddressFamily.InterNetworkV6; + var maxPrefixLength = isV6 ? 128 : 32; + int prefixLength; + + if (slash < 0) + { + prefixLength = maxPrefixLength; + } + else if (!int.TryParse(cidr[(slash + 1)..], out prefixLength) || + prefixLength < 0 || prefixLength > maxPrefixLength) + { + return false; + } + + if (!isV6) + { + prefixLength += 96; // 32 -> 128 + } + + Span bytes = stackalloc byte[16]; + ip.WriteMappedIPv6To(bytes); + + min = Utility.CreateCidrAddress(bytes, prefixLength, false); + max = Utility.CreateCidrAddress(bytes, prefixLength, true); + return true; + } + + /// Extracts the big-endian uint of an address. + public static bool TryV4(IPAddress ip, out uint v) + { + Span b = stackalloc byte[4]; + if (ip.TryWriteBytes(b, out var n) && n == 4) + { + v = ((uint)b[0] << 24) | ((uint)b[1] << 16) | ((uint)b[2] << 8) | b[3]; + return true; + } + + v = 0; + return false; + } + + /// + /// Extracts the embedded v4 uint from a v4-mapped-v6 address directly from the mapped bytes, + /// avoiding the allocation of . + /// + public static bool TryMappedV4(IPAddress ip, out uint v) + { + Span b = stackalloc byte[16]; + if (ip.TryWriteBytes(b, out var n) && n == 16) + { + v = ((uint)b[12] << 24) | ((uint)b[13] << 16) | ((uint)b[14] << 8) | b[15]; + return true; + } + + v = 0; + return false; + } + + /// Parses a dotted-quad IPv4 literal into a big-endian uint. Allocation-free, strict. + public static bool TryParseV4(ReadOnlySpan s, out uint v) + { + v = 0; + uint acc = 0; + int octet = 0, digits = 0, dots = 0; + for (var i = 0; i < s.Length; i++) + { + var c = s[i]; + if (c == '.') + { + if (digits == 0 || octet > 255) + { + return false; + } + + acc = (acc << 8) | (uint)octet; + dots++; + octet = 0; + digits = 0; + } + else if (c is >= '0' and <= '9') + { + octet = octet * 10 + (c - '0'); + if (++digits > 3) + { + return false; + } + } + else + { + return false; + } + } + + if (dots != 3 || digits == 0 || octet > 255) + { + return false; + } + + v = (acc << 8) | (uint)octet; + return true; + } + + /// + /// UTF-8/ASCII byte overload of , mirroring its + /// validation exactly so the blocklist can parse dotted-quads straight from file bytes with no + /// per-line string allocation. + /// + public static bool TryParseV4(ReadOnlySpan s, out uint v) + { + v = 0; + uint acc = 0; + int octet = 0, digits = 0, dots = 0; + for (var i = 0; i < s.Length; i++) + { + var c = s[i]; + if (c == (byte)'.') + { + if (digits == 0 || octet > 255) + { + return false; + } + + acc = (acc << 8) | (uint)octet; + dots++; + octet = 0; + digits = 0; + } + else if (c is >= (byte)'0' and <= (byte)'9') + { + octet = octet * 10 + (c - '0'); + if (++digits > 3) + { + return false; + } + } + else + { + return false; + } + } + + if (dots != 3 || digits == 0 || octet > 255) + { + return false; + } + + v = (acc << 8) | (uint)octet; + return true; + } +} diff --git a/Projects/Server/Utilities/NetworkUtilities.cs b/Projects/Server/Utilities/NetworkUtilities.cs index 7d54ddf5e..deeffb66f 100644 --- a/Projects/Server/Utilities/NetworkUtilities.cs +++ b/Projects/Server/Utilities/NetworkUtilities.cs @@ -1,6 +1,7 @@ +using System; using System.Net; using System.Net.Sockets; -using Server.Network; +using Server.Collections; namespace Server; @@ -14,36 +15,42 @@ public static class NetworkUtilities _ => false }; - private static readonly IFirewallEntry[] _privateNetworkV4 = - [ - new CidrFirewallEntry("127.0.0.1/8"), - new CidrFirewallEntry("192.168.0.0/16"), - new CidrFirewallEntry("10.0.0.0/8"), - new CidrFirewallEntry("172.16.0.0/12"), - new CidrFirewallEntry("169.254.0.0/16"), - new CidrFirewallEntry("100.64.0.0/10") - ]; + // These are constant reserved ranges, not firewall entries -- they only ever answer "is this address + // in one of these blocks?", which is exactly what SortedRangeIndex is for. Building them through the + // firewall entry types was a convenience that made core depend on the firewall for something that has + // nothing to do with banning. + private static readonly SortedRangeIndex _privateNetworkV4 = BuildIndex( + "127.0.0.1/8", + "192.168.0.0/16", + "10.0.0.0/8", + "172.16.0.0/12", + "169.254.0.0/16", + "100.64.0.0/10" + ); - private static readonly IFirewallEntry[] _privateNetworkV6 = - [ - new CidrFirewallEntry("fc00::/7"), - new CidrFirewallEntry("fe80::/10") - ]; + private static readonly SortedRangeIndex _privateNetworkV6 = BuildIndex( + "fc00::/7", + "fe80::/10" + ); - public static bool IsPrivateNetworkV4(this IPAddress ip) + private static SortedRangeIndex BuildIndex(params ReadOnlySpan cidrs) { - for (var i = 0; i < _privateNetworkV4.Length; i++) + var ranges = new SortedRangeIndex.Range[cidrs.Length]; + for (var i = 0; i < cidrs.Length; i++) { - if (_privateNetworkV4[i].IsBlocked(ip)) + if (!IPAddressUtility.TryParseCidrRange(cidrs[i], out var min, out var max)) { - return true; + throw new ArgumentException($"Invalid reserved-network CIDR \"{cidrs[i]}\""); } + + ranges[i] = new SortedRangeIndex.Range(min, max); } - return false; + Array.Sort(ranges, SortedRangeIndex.ByMin); + return SortedRangeIndex.Build(ranges); } - public static bool IsPrivateNetworkV6(this IPAddress ip) => - _privateNetworkV6[0].IsBlocked(ip) || - _privateNetworkV6[1].IsBlocked(ip); + public static bool IsPrivateNetworkV4(this IPAddress ip) => _privateNetworkV4.Contains(ip.ToUInt128()); + + public static bool IsPrivateNetworkV6(this IPAddress ip) => _privateNetworkV6.Contains(ip.ToUInt128()); } diff --git a/Projects/Server/Utilities/Utility.cs b/Projects/Server/Utilities/Utility.cs index 4f33220aa..f3786d834 100644 --- a/Projects/Server/Utilities/Utility.cs +++ b/Projects/Server/Utilities/Utility.cs @@ -108,45 +108,6 @@ public static partial class Utility } } - // Converts an IPAddress to a UInt128 in IPv6 format - public static UInt128 ToUInt128(this IPAddress ip) - { - if (ip.AddressFamily == AddressFamily.InterNetwork && !ip.IsIPv4MappedToIPv6) - { - Span integer = stackalloc byte[4]; - return !ip.TryWriteBytes(integer, out _) - ? (UInt128)0 - : new UInt128(0, 0xFFFF00000000UL | BinaryPrimitives.ReadUInt32BigEndian(integer)); - } - - Span bytes = stackalloc byte[16]; - if (!ip.TryWriteBytes(bytes, out _)) - { - return 0; - } - - var high = BinaryPrimitives.ReadUInt64BigEndian(bytes[..8]); - var low = BinaryPrimitives.ReadUInt64BigEndian(bytes.Slice(8, 8)); - - return new UInt128(high, low); - } - - // Converts a UInt128 in IPv6 format to an IPAddress - public static IPAddress ToIpAddress(this UInt128 value, bool mapToIpv6 = false) - { - // IPv4 mapped IPv6 address - if (!mapToIpv6 && value >= 0xFFFF00000000UL && value <= 0xFFFFFFFFFFFFUL) - { - var newAddress = IPAddress.HostToNetworkOrder((int)value); - return new IPAddress(unchecked((uint)newAddress)); - } - - Span bytes = stackalloc byte[16]; // 128 bits for IPv6 address - ((IBinaryInteger)value).WriteBigEndian(bytes); - - return new IPAddress(bytes); - } - [MethodImpl(MethodImplOptions.AggressiveInlining)] public static UInt128 CreateCidrAddress(ReadOnlySpan bytes, int prefixLength, bool isMax) { diff --git a/Projects/UOContent.Tests/Fixtures/TestServerInitializer.cs b/Projects/UOContent.Tests/Fixtures/TestServerInitializer.cs index 076f88141..640c05837 100644 --- a/Projects/UOContent.Tests/Fixtures/TestServerInitializer.cs +++ b/Projects/UOContent.Tests/Fixtures/TestServerInitializer.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.IO; using System.Reflection; using System.Threading; @@ -61,6 +61,15 @@ internal static class TestServerInitializer AssemblyHandler.LoadAssemblies(["Server.dll", "UOContent.dll"]); SkillsInfo.Configure(); + + // Seed the loop clock as Main.cs does before the Configure sweep; otherwise Core.Now is + // DateTime.MinValue for the whole test host. + Core._now = DateTime.UtcNow; + + // Timer wheel must exist before NetState.Configure(), which schedules a recurring + // sweep via Timer.DelayCall (matches production ordering in Main.cs: Timer.Init runs + // before AssemblyHandler.Invoke("Configure")). + Timer.Init(0); Server.Network.NetState.Configure(); TestMapDefinitions.ConfigureTestMapDefinitions(); @@ -91,7 +100,6 @@ internal static class TestServerInitializer } World.Configure(); - Timer.Init(0); RaceDefinitions.Configure(); MovementImpl.Configure(); PathFollower.Configure(); diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistConfigurationTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistConfigurationTests.cs new file mode 100644 index 000000000..6464a1bed --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistConfigurationTests.cs @@ -0,0 +1,65 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistConfigurationTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Text.Json; +using Server.Json; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans.Blocklist; + +public class BlocklistConfigurationTests +{ + // Locks the JsonConfig casing contract: JsonConfig's options are case-SENSITIVE, so every settings + // member must carry an explicit [JsonPropertyName("camelCase")] or it silently binds nothing. + [Fact] + public void BlocklistSettings_RoundTripsThroughJsonConfig() + { + var original = new BlocklistSettings + { + File = "D:/shared/ip-blocklist.txt", + ReloadInterval = TimeSpan.FromMinutes(5), + ReportHits = false, + BanDuration = TimeSpan.FromHours(2), + PromoteSuppression = TimeSpan.FromSeconds(30) + }; + + var json = JsonConfig.Serialize(original); + + Assert.Contains("\"file\"", json); + Assert.Contains("\"reloadInterval\"", json); + Assert.Contains("\"reportHits\"", json); + Assert.Contains("\"banDuration\"", json); + Assert.Contains("\"promoteSuppression\"", json); + + var restored = JsonSerializer.Deserialize(json, JsonConfig.DefaultOptions); + + Assert.NotNull(restored); + Assert.Equal(original.File, restored.File); + Assert.Equal(original.ReloadInterval, restored.ReloadInterval); + Assert.Equal(original.ReportHits, restored.ReportHits); + Assert.Equal(original.BanDuration, restored.BanDuration); + Assert.Equal(original.PromoteSuppression, restored.PromoteSuppression); + } + + // The generator (tools/Export-IpBlocklist.ps1) writes to this path by default; if one side moves + // without the other, a shard silently enforces nothing. + [Fact] + public void Default_file_matches_the_generator_output_path() + { + Assert.Equal("Configuration/ip-blocklist.txt", new BlocklistSettings().File); + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFileTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFileTests.cs new file mode 100644 index 000000000..1b7a7772c --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFileTests.cs @@ -0,0 +1,85 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistFileTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; +using System.Net; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans.Blocklist; + +public class BlocklistFileTests +{ + private static string WriteTemp(string content) + { + var p = Path.Combine(Path.GetTempPath(), "bl-" + Guid.NewGuid().ToString("N") + ".txt"); + File.WriteAllText(p, content); + return p; + } + + [Fact] + public void Reads_header_generated_and_count() + { + var p = WriteTemp("# modernuo-blocklist v1 generated=2026-07-21T09:24:23Z count=2\n1.2.3.4\n5.6.7.0/24\n"); + Assert.True(BlocklistFile.TryReadHeader(p, out var h)); + Assert.True(h.Present); + Assert.Equal("2026-07-21T09:24:23Z", h.Generated); + Assert.Equal(2, h.Count); + File.Delete(p); + } + + [Fact] + public void Missing_file_reports_absent_and_loads_empty() + { + var p = Path.Combine(Path.GetTempPath(), "does-not-exist-" + Guid.NewGuid().ToString("N")); + Assert.False(BlocklistFile.TryReadHeader(p, out var h)); + Assert.False(h.Present); + var snap = BlocklistFile.Load(p, out _, out _); + Assert.False(snap.IsBanned(IPAddress.Parse("1.2.3.4"))); + } + + // Pins the exact line tools/Export-IpBlocklist.ps1 emits: the producer adds informational tokens the + // reader doesn't know about, and the reload detector breaks silently if generated= stops being read. + [Fact] + public void Reads_generator_header_with_extra_tokens() + { + var p = WriteTemp( + "# modernuo-blocklist generated=2026-07-25T18:03:11Z count=12442 ipv4=7868 cidr=4574 feeds=2\n" + + "2.181.183.77\n223.169.0.0/16\n" + ); + + Assert.True(BlocklistFile.TryReadHeader(p, out var h)); + Assert.Equal("2026-07-25T18:03:11Z", h.Generated); + Assert.Equal(12442, h.Count); + + var snap = BlocklistFile.Load(p, out var parsed, out var skipped); + Assert.Equal(2, parsed); + Assert.Equal(0, skipped); + Assert.True(snap.IsBanned(IPAddress.Parse("2.181.183.77"))); + Assert.True(snap.IsBanned(IPAddress.Parse("223.169.4.9"))); + File.Delete(p); + } + + [Fact] + public void Load_parses_body() + { + var p = WriteTemp("# generated=x count=1\n8.8.8.0/24\n"); + var snap = BlocklistFile.Load(p, out var parsed, out _); + Assert.Equal(1, parsed); + Assert.True(snap.IsBanned(IPAddress.Parse("8.8.8.8"))); + File.Delete(p); + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFilterTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFilterTests.cs new file mode 100644 index 000000000..4f3ee5ad1 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistFilterTests.cs @@ -0,0 +1,94 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistFilterTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Net; +using System.Text; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans.Blocklist; + +// No [Collection] and no static reset hook: the filter is an instance, so each test owns its own +// snapshot and promote-guard. That is the point of it no longer being a static class. +public class BlocklistFilterTests +{ + private static BlocklistFilter WithList(string list, bool reportHits = true, long suppressionMs = 5000) + { + var filter = new BlocklistFilter(); + filter.LoadForTesting( + BlocklistSnapshot.Build(Encoding.ASCII.GetBytes(list), out _, out _), + reportHits, + suppressionMs + ); + + return filter; + } + + [Fact] + public void Listed_address_is_denied_and_reported_once_per_window() + { + var filter = WithList("1.2.3.4"); + var ip = IPAddress.Parse("1.2.3.4"); + + var deny1 = filter.Evaluate(ip, 1000, out var report1); + var deny2 = filter.Evaluate(ip, 1500, out var report2); + var deny3 = filter.Evaluate(ip, 1000 + 5001, out var report3); + + Assert.True(deny1); + Assert.True(report1); + + Assert.True(deny2); + Assert.False(report2); // denied again, but promotion suppressed inside the window + + Assert.True(deny3); + Assert.True(report3); // window elapsed, promotion may be retried + } + + [Fact] + public void Unlisted_address_passes() + { + var filter = WithList("1.2.3.4"); + + Assert.False(filter.Evaluate(IPAddress.Parse("9.9.9.9"), 1, out var report)); + Assert.False(report); + } + + [Fact] + public void Cidr_membership_is_honored() + { + var filter = WithList("10.20.30.0/24"); + + Assert.True(filter.Evaluate(IPAddress.Parse("10.20.30.255"), 1, out _)); + Assert.False(filter.Evaluate(IPAddress.Parse("10.20.31.0"), 1, out _)); + } + + [Fact] + public void ReportHits_disabled_still_denies_but_never_promotes() + { + var filter = WithList("1.2.3.4", reportHits: false); + + Assert.True(filter.Evaluate(IPAddress.Parse("1.2.3.4"), 1, out var report)); + Assert.False(report); + } + + [Fact] + public void Unconfigured_filter_denies_nothing() + { + var filter = new BlocklistFilter(); + + Assert.Equal(0, filter.Count); + Assert.False(filter.ShouldDeny(IPAddress.Parse("8.8.8.8"))); + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistSnapshotTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistSnapshotTests.cs new file mode 100644 index 000000000..abdac1d11 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/BlocklistSnapshotTests.cs @@ -0,0 +1,98 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistSnapshotTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Net; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans.Blocklist; + +public class BlocklistSnapshotTests +{ + private static BlocklistSnapshot Build(params string[] lines) => + BlocklistSnapshot.Build(System.Text.Encoding.ASCII.GetBytes(string.Join('\n', lines)), out _, out _); + + [Fact] + public void Single_ip_is_matched() + { + var s = Build("1.2.3.4"); + Assert.True(s.IsBanned(IPAddress.Parse("1.2.3.4"))); + Assert.False(s.IsBanned(IPAddress.Parse("1.2.3.5"))); + } + + [Fact] + public void Cidr_contains_and_excludes_boundaries() + { + var s = Build("10.0.0.0/24"); + Assert.True(s.IsBanned(IPAddress.Parse("10.0.0.0"))); + Assert.True(s.IsBanned(IPAddress.Parse("10.0.0.255"))); + Assert.False(s.IsBanned(IPAddress.Parse("10.0.1.0"))); + Assert.False(s.IsBanned(IPAddress.Parse("9.255.255.255"))); + } + + [Fact] + public void Comments_blanks_and_garbage_are_skipped_not_thrown() + { + var s = BlocklistSnapshot.Build( + System.Text.Encoding.ASCII.GetBytes( + string.Join('\n', "# header generated=x", "", "not-an-ip", "1.2.3.4", "::1", "5.6.7.0/24")), + out var parsed, out var skipped); + Assert.Equal(3, parsed); // 1.2.3.4 + ::1 (valid loopback) + 5.6.7.0/24 + Assert.True(skipped >= 1); // "not-an-ip"; blank/comment lines are silently skipped, not counted + Assert.True(s.IsBanned(IPAddress.Parse("5.6.7.200"))); + } + + [Fact] + public void Empty_snapshot_matches_nothing() + { + Assert.False(BlocklistSnapshot.Empty.IsBanned(IPAddress.Parse("1.2.3.4"))); + } + + [Fact] + public void Ipv6_single_and_cidr_are_matched() + { + var s = Build("2001:db8::1", "2001:db8:1::/48"); + Assert.True(s.IsBanned(IPAddress.Parse("2001:db8::1"))); + Assert.True(s.IsBanned(IPAddress.Parse("2001:db8:1::abcd"))); + Assert.False(s.IsBanned(IPAddress.Parse("2001:db8:2::1"))); + } + + [Fact] + public void Ipv4_mapped_ipv6_is_normalized_to_v4() + { + var s = Build("1.2.3.4"); + Assert.True(s.IsBanned(IPAddress.Parse("::ffff:1.2.3.4"))); // must not bypass the v4 set + } + + [Fact] + public void Nested_cidr_intervals_are_coalesced() + { + // A /32 nested inside a /24: InRange's binary search only inspects the + // rightmost interval starting <= ip, so without coalescing an IP inside + // the /24 but outside the /32 would land on the /32 and wrongly pass. + var s = Build("10.0.0.0/24", "10.0.0.5/32"); + Assert.True(s.IsBanned(IPAddress.Parse("10.0.0.100"))); // inside /24, outside /32 + Assert.True(s.IsBanned(IPAddress.Parse("10.0.0.5"))); // the nested /32 itself + Assert.False(s.IsBanned(IPAddress.Parse("10.0.1.0"))); // genuinely outside both + } + + [Fact] + public void Overlapping_cidr_intervals_are_coalesced() + { + var s = Build("10.0.0.0/25", "10.0.0.64/25"); + Assert.True(s.IsBanned(IPAddress.Parse("10.0.0.100"))); // covered by the second /25 + Assert.False(s.IsBanned(IPAddress.Parse("10.0.0.200"))); // outside both + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/PromotedGuardTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/PromotedGuardTests.cs new file mode 100644 index 000000000..7c19b8d38 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/Blocklist/PromotedGuardTests.cs @@ -0,0 +1,46 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: PromotedGuardTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using Server.Network.Bans; +using Xunit; + +namespace Server.Tests.Network.Bans.Blocklist; + +public class PromotedGuardTests +{ + [Fact] + public void First_mark_true_then_suppressed_until_ttl() + { + var g = new PromotedGuard(); + Assert.True(g.TryMark((UInt128)42, 1000, 5000)); + Assert.False(g.TryMark((UInt128)42, 2000, 5000)); // within TTL + Assert.True(g.TryMark((UInt128)42, 6001, 5000)); // expired → re-mark + } + + [Fact] + public void Sweep_removes_expired_entries_allowing_remark() + { + var g = new PromotedGuard(); + Assert.True(g.TryMark((UInt128)7, 0, 1000)); + Assert.False(g.TryMark((UInt128)7, 500, 1000)); // still within TTL + + g.Sweep(500); // not yet expired, sweep should not remove it + Assert.False(g.TryMark((UInt128)7, 999, 1000)); + + g.Sweep(1001); // now expired, sweep removes it + Assert.True(g.TryMark((UInt128)7, 1002, 1000)); // fresh mark, not "still marked" + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecAlertClientTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecAlertClientTests.cs new file mode 100644 index 000000000..f8a990f25 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecAlertClientTests.cs @@ -0,0 +1,51 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecAlertClientTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Net; +using Server.Network.Bans.CrowdSec; +using Xunit; + +namespace Server.Tests.Network.Bans; + +public class CrowdSecAlertClientTests +{ + [Fact] + public void BuildDeleteQuery_EscapesOrigin() + { + // origin is operator-controlled config (crowdsec.json), so it must be treated as untrusted + // input going into the URL, same as any other interpolated value. + var query = CrowdSecAlertClient.BuildDeleteQuery("modern uo/test&x=1", IPAddress.Parse("1.2.3.4")); + + Assert.Equal("/v1/decisions?origin=modern%20uo%2Ftest%26x%3D1&ip=1.2.3.4", query); + } + + [Fact] + public void BuildDeleteQuery_PlainOrigin_Ipv4() + { + var query = CrowdSecAlertClient.BuildDeleteQuery("modernuo", IPAddress.Parse("192.168.1.1")); + + Assert.Equal("/v1/decisions?origin=modernuo&ip=192.168.1.1", query); + } + + [Fact] + public void BuildDeleteQuery_Ipv6_IsEscaped() + { + // IPv6 textual form contains ':', which Uri.EscapeDataString percent-encodes like any other + // reserved character — confirms the escaping is applied uniformly, not just for IPv4. + var query = CrowdSecAlertClient.BuildDeleteQuery("modernuo", IPAddress.Parse("2001:db8::1")); + + Assert.Equal("/v1/decisions?origin=modernuo&ip=2001%3Adb8%3A%3A1", query); + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecConfigurationTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecConfigurationTests.cs new file mode 100644 index 000000000..dd331ac6d --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecConfigurationTests.cs @@ -0,0 +1,82 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecConfigurationTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Text.Json; +using Server.Json; +using Server.Network.Bans.CrowdSec; +using Xunit; + +namespace Server.Tests.Network.Bans; + +public class CrowdSecConfigurationTests +{ + // Locks the JsonConfig casing/converter contract: JsonConfig's options are case-SENSITIVE, so + // every settings member must carry an explicit [JsonPropertyName("camelCase")] or it silently + // binds nothing. These tests round-trip through the exact options the loader uses. + + [Fact] + public void CrowdSecSettings_RoundTripsThroughJsonConfig() + { + var original = new CrowdSecSettings + { + LapiUrl = "http://10.0.0.5:9090", + MachineId = "shard", + Password = "secret", + Origin = "modernuo", + ManualBanDuration = TimeSpan.FromHours(168), + FlushInterval = TimeSpan.FromSeconds(1), + MaxQueue = 10000 + }; + + var json = JsonConfig.Serialize(original); + + // camelCase property names must be present (not PascalCase) or the case-sensitive reader binds nothing. + Assert.Contains("\"lapiUrl\"", json); + Assert.Contains("\"machineId\"", json); + Assert.Contains("\"password\"", json); + Assert.Contains("\"origin\"", json); + Assert.Contains("\"manualBanDuration\"", json); + Assert.Contains("\"flushInterval\"", json); + Assert.Contains("\"maxQueue\"", json); + + var restored = JsonSerializer.Deserialize(json, JsonConfig.DefaultOptions); + + Assert.NotNull(restored); + Assert.Equal(original.LapiUrl, restored.LapiUrl); + Assert.Equal(original.MachineId, restored.MachineId); + Assert.Equal(original.Password, restored.Password); + Assert.Equal(original.Origin, restored.Origin); + Assert.Equal(original.ManualBanDuration, restored.ManualBanDuration); // TimeSpan survives + Assert.Equal(original.FlushInterval, restored.FlushInterval); // TimeSpan survives + Assert.Equal(original.MaxQueue, restored.MaxQueue); + Assert.True(restored.ReportingEnabled); + } + + [Fact] + public void CrowdSecSettings_Defaults_AreReportingDisabledLocalLoopback() + { + var settings = new CrowdSecSettings(); + + Assert.Equal("http://127.0.0.1:8080", settings.LapiUrl); + Assert.Equal("", settings.MachineId); + Assert.Equal("", settings.Password); + Assert.Equal("modernuo", settings.Origin); + Assert.Equal(TimeSpan.FromHours(168), settings.ManualBanDuration); + Assert.Equal(TimeSpan.FromSeconds(1), settings.FlushInterval); + Assert.Equal(10000, settings.MaxQueue); + Assert.False(settings.ReportingEnabled); // empty machineId/password => inert + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecReporterTests.cs b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecReporterTests.cs new file mode 100644 index 000000000..a8daf3c44 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Bans/CrowdSecReporterTests.cs @@ -0,0 +1,229 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecReporterTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading; +using System.Threading.Tasks; +using Server.Network.Bans.CrowdSec; +using Xunit; + +namespace Server.Tests.Network.Bans; + +public class CrowdSecReporterTests +{ + private static CrowdSecSettings Settings() => new() + { + MachineId = "shard", + Password = "secret", + Origin = "modernuo", + ManualBanDuration = TimeSpan.FromHours(168) + }; + + [Fact] + public void BuildAlerts_DedupsByIp() + { + var now = DateTime.UnixEpoch; + var items = new List + { + new(IPAddress.Parse("1.1.1.1"), TimeSpan.FromHours(1), "rate-limit", false), + new(IPAddress.Parse("1.1.1.1"), TimeSpan.FromHours(1), "rate-limit", false), + new(IPAddress.Parse("2.2.2.2"), TimeSpan.FromHours(1), "rate-limit", false) + }; + + var alerts = CrowdSecReporter.BuildAlerts(items, Settings(), now); + + Assert.Equal(2, alerts.Count); + Assert.All(alerts, a => Assert.Single(a.Decisions)); + Assert.Contains(alerts, a => a.Source.Value == "1.1.1.1"); + Assert.Contains(alerts, a => a.Source.Value == "2.2.2.2"); + } + + [Fact] + public void BuildAlerts_ScenarioFromReason_OriginFromSettings() + { + var alerts = CrowdSecReporter.BuildAlerts( + [new(IPAddress.Parse("3.3.3.3"), TimeSpan.FromHours(1), "manual", false)], + Settings(), + DateTime.UnixEpoch); + + var decision = Assert.Single(alerts).Decisions[0]; + Assert.Equal("modernuo/manual", alerts[0].Scenario); + Assert.Equal("modernuo", decision.Origin); + Assert.Equal("ban", decision.Type); + Assert.Equal("Ip", decision.Scope); + Assert.Equal("3.3.3.3", decision.Value); + } + + [Fact] + public void BuildAlerts_ScenarioFromReason_Blocklist() + { + var alerts = CrowdSecReporter.BuildAlerts( + [new(IPAddress.Parse("5.5.5.5"), TimeSpan.FromHours(1), "blocklist", false)], + Settings(), + DateTime.UnixEpoch); + + var decision = Assert.Single(alerts).Decisions[0]; + Assert.Equal("modernuo/blocklist", alerts[0].Scenario); + Assert.Equal("modernuo/blocklist", decision.Scenario); + } + + [Fact] + public void FormatDuration_UsesSeconds_FloorsAtOne() + { + Assert.Equal("3600s", CrowdSecReporter.FormatDuration(TimeSpan.FromHours(1))); + Assert.Equal("1s", CrowdSecReporter.FormatDuration(TimeSpan.Zero)); + Assert.Equal("1s", CrowdSecReporter.FormatDuration(TimeSpan.FromMilliseconds(10))); + } + + [Fact] + public void Report_WhenQueueFull_DropsAndCounts() + { + var reporter = new CrowdSecReporter(new NullAlertClient(), new CrowdSecSettings + { + MachineId = "shard", + Password = "secret", + MaxQueue = 2 + }); + // Do NOT Start() the drain — so the queue fills and overflows deterministically. + + for (var i = 0; i < 10; i++) + { + reporter.Report(IPAddress.Parse("4.4.4." + i), TimeSpan.FromHours(1), "rate-limit"); + } + + Assert.True(reporter.DroppedCount >= 8); + } + + // Stop() without a prior Start() drives FlushRemainingOnStop() synchronously (no drain task, no + // Task.Delay backoff involved), so this is deterministic — no wall-clock timing dependency. + [Fact] + public void Stop_FlushesQueuedReports_ViaClient() + { + var client = new RecordingAlertClient(); + var reporter = new CrowdSecReporter(client, Settings()); + + reporter.Report(IPAddress.Parse("6.6.6.6"), TimeSpan.FromHours(1), "rate-limit"); + reporter.Stop(); + + var posted = Assert.Single(client.Posted); + Assert.Equal("6.6.6.6", Assert.Single(posted).Source.Value); + Assert.Equal(0, reporter.SendFailureCount); + } + + [Fact] + public void Stop_FlushesQueuedRetracts_ViaClient() + { + var client = new RecordingAlertClient(); + var reporter = new CrowdSecReporter(client, Settings()); + + reporter.Retract(IPAddress.Parse("8.8.8.8")); + reporter.Stop(); + + Assert.Equal(IPAddress.Parse("8.8.8.8"), Assert.Single(client.Deleted)); + Assert.Equal(0, reporter.SendFailureCount); + } + + [Fact] + public void Stop_WhenFlushSendFails_CountsSendFailure() + { + var reporter = new CrowdSecReporter(new ThrowingAlertClient(), Settings()); + + reporter.Report(IPAddress.Parse("7.7.7.7"), TimeSpan.FromHours(1), "rate-limit"); + reporter.Stop(); + + Assert.Equal(1, reporter.SendFailureCount); + } + + [Fact] + public void Stop_WithEmptyQueue_DoesNotInvokeClientOrFail() + { + var client = new RecordingAlertClient(); + var reporter = new CrowdSecReporter(client, Settings()); + + reporter.Stop(); + + Assert.Empty(client.Posted); + Assert.Equal(0, reporter.SendFailureCount); + } + + /// + /// The drain task must track the loop's lifetime, not just its first await — a ValueTask-returning + /// drain loop passed to Task.Run yields a Task<ValueTask> that completes immediately, which makes + /// Stop()'s drain-exited handshake a no-op. With an empty queue the loop parks on WaitToReadAsync, + /// so a correctly unwrapped task cannot win this race; a slow pool only under-detects. + /// + [Fact] + public async Task Start_DrainTaskSpansLoopLifetime_NotJustTheFirstAwait() + { + var reporter = new CrowdSecReporter(new NullAlertClient(), Settings()); + using var cts = new CancellationTokenSource(); + + reporter.Start(cts.Token); + + var drain = reporter.DrainTaskForTesting; + Assert.NotNull(drain); + + var first = await Task.WhenAny(drain, Task.Delay(TimeSpan.FromMilliseconds(500))); + Assert.False(ReferenceEquals(first, drain), "drain task completed while the loop was still running"); + + reporter.Stop(); + + Assert.True(drain.IsCompleted, "Stop() returned before the drain loop exited"); + } + + private sealed class NullAlertClient : ICrowdSecAlertClient + { + public ValueTask PostAlertsAsync(IReadOnlyList alerts, CancellationToken token) => + ValueTask.CompletedTask; + + public ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token) => + ValueTask.CompletedTask; + + public void Dispose() { } + } + + private sealed class RecordingAlertClient : ICrowdSecAlertClient + { + public List> Posted { get; } = []; + public List Deleted { get; } = []; + + public ValueTask PostAlertsAsync(IReadOnlyList alerts, CancellationToken token) + { + Posted.Add(alerts); + return ValueTask.CompletedTask; + } + + public ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token) + { + Deleted.Add(ip); + return ValueTask.CompletedTask; + } + + public void Dispose() { } + } + + private sealed class ThrowingAlertClient : ICrowdSecAlertClient + { + public ValueTask PostAlertsAsync(IReadOnlyList alerts, CancellationToken token) => + throw new InvalidOperationException("simulated LAPI outage"); + + public ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token) => + ValueTask.CompletedTask; + + public void Dispose() { } + } +} diff --git a/Projects/Server.Tests/Tests/Network/Firewall/FirewallEntryTests.cs b/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallEntryTests.cs similarity index 100% rename from Projects/Server.Tests/Tests/Network/Firewall/FirewallEntryTests.cs rename to Projects/UOContent.Tests/Tests/Network/Firewall/FirewallEntryTests.cs diff --git a/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallPersistenceTests.cs b/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallPersistenceTests.cs new file mode 100644 index 000000000..01691e223 --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallPersistenceTests.cs @@ -0,0 +1,72 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: FirewallPersistenceTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Net; +using Server.Network; +using Xunit; + +namespace Server.Tests.Network.Firewall; + +[Collection("Sequential UOContent Tests")] +public class FirewallPersistenceTests +{ + [Fact] + public void ToSettings_RoundTrips_PermanentAndTtl() + { + Server.Network.Firewall.ResetForTesting(); + Server.Network.Firewall.Add(new SingleIpFirewallEntry(IPAddress.Parse("1.2.3.4"))); + Server.Network.Firewall.Add(new SingleIpFirewallEntry(IPAddress.Parse("2.2.2.2")), TimeSpan.FromHours(1)); + + var settings = Server.Network.Firewall.ToSettings(); + + Server.Network.Firewall.ResetForTesting(); + Server.Network.Firewall.LoadFrom(settings); + + Assert.True(Server.Network.Firewall.IsBlocked(IPAddress.Parse("1.2.3.4"))); + Assert.True(Server.Network.Firewall.IsBlocked(IPAddress.Parse("2.2.2.2"))); + } + + [Fact] + public void LoadFrom_SkipsAlreadyExpired() + { + Server.Network.Firewall.ResetForTesting(); + var settings = new FirewallSettings + { + Entries = + [ + // Core.Now, matching the clock LoadFrom compares against. + new FirewallEntryRecord { Value = "9.9.9.9", Expires = Core.Now.AddHours(-1) } + ] + }; + + Server.Network.Firewall.LoadFrom(settings); + + Assert.False(Server.Network.Firewall.IsBlocked(IPAddress.Parse("9.9.9.9"))); + } + + [Fact] + public void ToSettings_OmitsExpiryForPermanent() + { + Server.Network.Firewall.ResetForTesting(); + Server.Network.Firewall.Add(new SingleIpFirewallEntry(IPAddress.Parse("1.2.3.4"))); + + var settings = Server.Network.Firewall.ToSettings(); + + Assert.Single(settings.Entries); + Assert.Null(settings.Entries[0].Expires); + Assert.Equal("1.2.3.4", settings.Entries[0].Value); + } +} diff --git a/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallTests.cs b/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallTests.cs new file mode 100644 index 000000000..fabf6d4ba --- /dev/null +++ b/Projects/UOContent.Tests/Tests/Network/Firewall/FirewallTests.cs @@ -0,0 +1,89 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: FirewallTests.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Net; +using Server.Network; +using Xunit; + +namespace Server.Tests.Network.Firewall; + +[Collection("Sequential UOContent Tests")] +public class FirewallTests +{ + private static IPAddress Ip(string s) => IPAddress.Parse(s); + + [Fact] + public void Add_ThenIsBlocked_SingleIp() + { + Server.Network.Firewall.ResetForTesting(); + Assert.True(Server.Network.Firewall.Add(new SingleIpFirewallEntry(Ip("1.2.3.4")))); + Assert.True(Server.Network.Firewall.IsBlocked(Ip("1.2.3.4"))); + Assert.False(Server.Network.Firewall.IsBlocked(Ip("1.2.3.5"))); + } + + [Fact] + public void Add_Range_BlocksInside_NotOutside() + { + Server.Network.Firewall.ResetForTesting(); + Server.Network.Firewall.Add(new CidrFirewallEntry(Ip("10.0.0.0"), Ip("10.0.0.255"))); + Assert.True(Server.Network.Firewall.IsBlocked(Ip("10.0.0.7"))); + Assert.False(Server.Network.Firewall.IsBlocked(Ip("10.0.1.0"))); + } + + [Fact] + public void Remove_Unblocks() + { + Server.Network.Firewall.ResetForTesting(); + var entry = new SingleIpFirewallEntry(Ip("1.2.3.4")); + Server.Network.Firewall.Add(entry); + Assert.True(Server.Network.Firewall.Remove(entry)); + Assert.False(Server.Network.Firewall.IsBlocked(Ip("1.2.3.4"))); + } + + [Fact] + public void ExpireEntries_RemovesExpired_KeepsPermanent() + { + Server.Network.Firewall.ResetForTesting(); + var permanent = new SingleIpFirewallEntry(Ip("1.1.1.1")); + var temporary = new SingleIpFirewallEntry(Ip("2.2.2.2")); + Server.Network.Firewall.Add(permanent); // no ttl + Server.Network.Firewall.Add(temporary, TimeSpan.FromMilliseconds(50)); // ttl + + Server.Network.Firewall.ExpireEntries(Core.TickCount + 100); // past the ttl + + Assert.True(Server.Network.Firewall.IsBlocked(Ip("1.1.1.1"))); + Assert.False(Server.Network.Firewall.IsBlocked(Ip("2.2.2.2"))); + } + + [Fact] + public void ToFirewallEntry_ParsesForms() + { + Assert.IsType(Server.Network.Firewall.ToFirewallEntry("1.2.3.4")); + Assert.IsType(Server.Network.Firewall.ToFirewallEntry("10.0.0.0/24")); + Assert.IsType(Server.Network.Firewall.ToFirewallEntry("10.0.0.0-10.0.0.255")); + Assert.Null(Server.Network.Firewall.ToFirewallEntry("not-an-ip")); + } + + [Fact] + public void ReadFirewallSet_SurfacesAddedEntries() + { + Server.Network.Firewall.ResetForTesting(); + var entry = new SingleIpFirewallEntry(Ip("1.2.3.4")); + Server.Network.Firewall.Add(entry); + + Server.Network.Firewall.ReadFirewallSet(set => Assert.Contains(entry, set)); + } +} diff --git a/Projects/UOContent/Commands/Generic/Commands/Commands.cs b/Projects/UOContent/Commands/Generic/Commands/Commands.cs index 81d85fc0a..b3fa95350 100644 --- a/Projects/UOContent/Commands/Generic/Commands/Commands.cs +++ b/Projects/UOContent/Commands/Generic/Commands/Commands.cs @@ -9,6 +9,7 @@ using Server.Items; using Server.Mobiles; using Server.Multis; using Server.Network; +using Server.Network.Bans; using Server.Spells; namespace Server.Commands.Generic @@ -1154,7 +1155,8 @@ namespace Server.Commands.Generic try { - AdminFirewall.Add(state.Address); + Firewall.Add(new SingleIpFirewallEntry(state.Address)); + BanChannel.Report(state.Address, TimeSpan.Zero, "manual"); AddResponse("They have been firewalled."); } catch (Exception ex) diff --git a/Projects/UOContent/Gumps/AdminGump.cs b/Projects/UOContent/Gumps/AdminGump.cs index 6e7c3ac23..1e788372e 100644 --- a/Projects/UOContent/Gumps/AdminGump.cs +++ b/Projects/UOContent/Gumps/AdminGump.cs @@ -11,6 +11,7 @@ using Server.Maps; using Server.Misc; using Server.Multis; using Server.Network; +using Server.Network.Bans; using Server.Prompts; using Server.Saves; using Server.Text; @@ -1743,7 +1744,8 @@ namespace Server.Gumps { for (var i = 0; i < a.LoginIPs.Length; ++i) { - AdminFirewall.Add(a.LoginIPs[i]); + Firewall.Add(new SingleIpFirewallEntry(a.LoginIPs[i])); + BanChannel.Report(a.LoginIPs[i], TimeSpan.Zero, "manual"); } notice = "All addresses in the list have been firewalled."; @@ -1767,7 +1769,13 @@ namespace Server.Gumps if (okay) { - AdminFirewall.Add(toFirewall); + var firewallEntry = Firewall.ToFirewallEntry(toFirewall); + Firewall.Add(firewallEntry); + + if (firewallEntry.MinIpAddress == firewallEntry.MaxIpAddress) + { + BanChannel.Report(firewallEntry.MinIpAddress.ToIpAddress(), TimeSpan.Zero, "manual"); + } notice = $"{toFirewall} : Added to firewall."; } @@ -3559,7 +3567,7 @@ namespace Server.Gumps IFirewallEntry firewallEntry; try { - firewallEntry = AdminFirewall.ToFirewallEntry(text); + firewallEntry = Firewall.ToFirewallEntry(text); } catch { @@ -3581,7 +3589,17 @@ namespace Server.Gumps $"{from.AccessLevel} {CommandLogging.Format(from)} firewalling {firewallEntry}" ); - AdminFirewall.Add(firewallEntry); + Firewall.Add(firewallEntry); + + if (firewallEntry.MinIpAddress == firewallEntry.MaxIpAddress) + { + BanChannel.Report( + firewallEntry.MinIpAddress.ToIpAddress(), + TimeSpan.Zero, + "manual" + ); + } + from.SendGump( new AdminGump( from, @@ -3620,7 +3638,13 @@ namespace Server.Gumps $"{from.AccessLevel} {CommandLogging.Format(from)} removing {m_State} from firewall list" ); - AdminFirewall.Remove(m_State); + Firewall.Remove(m_State as IFirewallEntry); + + if (m_State is IFirewallEntry fe && fe.MinIpAddress == fe.MaxIpAddress) + { + BanChannel.Retract(fe.MinIpAddress.ToIpAddress()); + } + from.SendGump( new AdminGump( from, diff --git a/Projects/UOContent/Misc/AdminFirewall.cs b/Projects/UOContent/Misc/AdminFirewall.cs deleted file mode 100644 index a4275d90c..000000000 --- a/Projects/UOContent/Misc/AdminFirewall.cs +++ /dev/null @@ -1,139 +0,0 @@ -using System; -using System.Buffers; -using System.IO; -using System.Net; -using System.Runtime.CompilerServices; -using Server.Logging; -using Server.Network; - -namespace Server; - -public static class AdminFirewall -{ - private static readonly ILogger logger = LogFactory.GetLogger(typeof(AdminFirewall)); - - private const string firewallConfigPath = "firewall.cfg"; - - public static void Configure() - { - if (File.Exists(firewallConfigPath)) - { - var searchValues = SearchValues.Create("*Xx?"); - - using var ip = new StreamReader(firewallConfigPath); - - while (ip.ReadLine() is { } line) - { - line = line.Trim(); - - if (line.Length == 0) - { - continue; - } - - if (line.AsSpan().ContainsAny(searchValues)) - { - logger.Warning("Legacy firewall entry \"{Entry}\" ignored", line); - continue; - } - - Add(ToFirewallEntry(line), false); - } - } - } - - [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static IFirewallEntry ToFirewallEntry(object entry) - { - return entry switch - { - IFirewallEntry firewallEntry => firewallEntry, - IPAddress address => new SingleIpFirewallEntry(address), - string s => ToFirewallEntry(s), - _ => null - }; - } - - public static IFirewallEntry ToFirewallEntry(string entry) - { - if (entry == null) - { - return null; - } - - try - { - var rangeSeparator = entry.IndexOf('-'); - if (rangeSeparator > -1) - { - return new CidrFirewallEntry( - IPAddress.Parse(entry.AsSpan(0, rangeSeparator)), - IPAddress.Parse(entry.AsSpan(rangeSeparator + 1)) - ); - } - - // CIDR notation - if (entry.IndexOf('/') > -1) - { - return new CidrFirewallEntry(entry); - } - - return new SingleIpFirewallEntry(entry); - } - catch - { - return null; - } - } - - public static bool Remove(object obj, bool save = true) - { - var entry = ToFirewallEntry(obj); - - if (entry == null) - { - return false; - } - - if (!Firewall.Remove(entry)) - { - return false; - } - - if (save) - { - Save(); - } - - return true; - } - - public static void Add(object obj) => Add(ToFirewallEntry(obj)); - - public static bool Add(IFirewallEntry entry, bool save = true) - { - if (!Firewall.Add(entry)) - { - return false; - } - - if (save) - { - Save(); - } - - return true; - } - - public static void Save() - { - Firewall.ReadFirewallSet(firewallSet => - { - using var op = new StreamWriter(firewallConfigPath); - foreach (var entry in firewallSet) - { - op.WriteLine(entry); - } - }); - } -} diff --git a/Projects/UOContent/Misc/Blocklist/BlocklistConfiguration.cs b/Projects/UOContent/Misc/Blocklist/BlocklistConfiguration.cs new file mode 100644 index 000000000..ca39bd53e --- /dev/null +++ b/Projects/UOContent/Misc/Blocklist/BlocklistConfiguration.cs @@ -0,0 +1,90 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistConfiguration.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; +using System.Text.Json.Serialization; +using Server.Json; + +namespace Server.Network.Bans; + +/// +/// Loads the from Configuration/blocklist.json (matching the +/// per-feature JSON config pattern used by AssistantConfiguration). Loaded once; a missing file +/// writes a template so operators have something to edit. +/// +public static class BlocklistConfiguration +{ + private const string _path = "Configuration/blocklist.json"; + + public static BlocklistSettings Settings { get; private set; } + + public static void Load() + { + var path = Path.Join(Core.BaseDirectory, _path); + + if (File.Exists(path)) + { + Settings = JsonConfig.Deserialize(path); + } + else + { + Settings = new BlocklistSettings(); + Save(); + } + } + + private static void Save() + { + JsonConfig.Serialize(Path.Join(Core.BaseDirectory, _path), Settings); + } +} + +/// +/// Bound configuration for . The filter is inert unless +/// points at a list that actually exists, so the shipped defaults are safe on a shard that never runs +/// the generator. +/// +public record BlocklistSettings +{ + /// + /// Path to the blocklist. A relative path resolves against ; an + /// absolute path is used as-is (handy when several shards share one generated list). Set to + /// "" to disable the gate entirely. Produce the file with tools/Export-IpBlocklist.ps1. + /// + [JsonPropertyName("file")] + public string File { get; set; } = "Configuration/ip-blocklist.txt"; + + /// How often the file is checked for changes. Reloads only happen when it actually changed. + [JsonPropertyName("reloadInterval")] + public TimeSpan ReloadInterval { get; set; } = TimeSpan.FromSeconds(60); + + /// Whether blocklist hits are contributed to the ban channel (the demand-paging promotion). + [JsonPropertyName("reportHits")] + public bool ReportHits { get; set; } = true; + + /// Duration reported for a blocklist-matched ban. + [JsonPropertyName("banDuration")] + public TimeSpan BanDuration { get; set; } = TimeSpan.FromHours(6); + + /// + /// How long the accept-path guard suppresses re-reporting a promoted address. This only needs to + /// bridge the gap until the OS bouncer picks up the promotion (seconds); after that the kernel drops + /// repeat traffic. Decoupled from so the guard doesn't have to remember + /// hours' worth of distinct addresses. + /// + [JsonPropertyName("promoteSuppression")] + public TimeSpan PromoteSuppression { get; set; } = TimeSpan.FromSeconds(60); +} diff --git a/Projects/UOContent/Misc/Blocklist/BlocklistFile.cs b/Projects/UOContent/Misc/Blocklist/BlocklistFile.cs new file mode 100644 index 000000000..a593a7035 --- /dev/null +++ b/Projects/UOContent/Misc/Blocklist/BlocklistFile.cs @@ -0,0 +1,86 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistFile.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; + +namespace Server.Network.Bans; + +public readonly record struct BlocklistHeader(string Generated, int Count, bool Present); + +/// Reads the versioned blocklist file: cheap header probe + full snapshot load. +public static class BlocklistFile +{ + public static bool TryReadHeader(string path, out BlocklistHeader header) + { + header = new BlocklistHeader(null, 0, false); + try + { + if (!File.Exists(path)) + { + return false; + } + using var reader = new StreamReader(path); + var first = reader.ReadLine(); + if (first == null) + { + header = new BlocklistHeader(null, 0, true); + return true; + } + string generated = null; + var count = 0; + if (first.StartsWith('#')) + { + var tokens = first.Split(' ', StringSplitOptions.RemoveEmptyEntries); + for (var i = 0; i < tokens.Length; i++) + { + var tok = tokens[i]; + if (tok.StartsWith("generated=", StringComparison.Ordinal)) + { + generated = tok["generated=".Length..]; + } + else if (tok.StartsWith("count=", StringComparison.Ordinal)) + { + int.TryParse(tok["count=".Length..], out count); + } + } + } + header = new BlocklistHeader(generated, count, true); + return true; + } + catch + { + return false; // treat as absent; caller keeps last-good / empty + } + } + + public static BlocklistSnapshot Load(string path, out int parsed, out int skipped) + { + parsed = 0; + skipped = 0; + try + { + if (!File.Exists(path)) + { + return BlocklistSnapshot.Empty; + } + return BlocklistSnapshot.Build(File.ReadAllBytes(path), out parsed, out skipped); + } + catch + { + return BlocklistSnapshot.Empty; + } + } +} diff --git a/Projects/UOContent/Misc/Blocklist/BlocklistFilter.cs b/Projects/UOContent/Misc/Blocklist/BlocklistFilter.cs new file mode 100644 index 000000000..aeca58477 --- /dev/null +++ b/Projects/UOContent/Misc/Blocklist/BlocklistFilter.cs @@ -0,0 +1,259 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistFilter.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; +using System.Net; +using System.Threading; +using System.Threading.Tasks; +using Server.Logging; + +namespace Server.Network.Bans; + +/// +/// Accept-path gate for a large, file-sourced IP blocklist, hydrated from the file a generator +/// (tools/Export-IpBlocklist.ps1) writes on a schedule. Holds an immutable snapshot swapped +/// atomically by an off-loop reload poll, so accept-path reads are lock-free. Inert when no file is +/// configured or present. +/// +/// +/// This is the demand-paging half of the design: an OS firewall cannot hold millions of entries on +/// Windows, so the millions live here and only addresses that actually connect are promoted to CrowdSec +/// (and from there to the OS firewall) through . +/// keeps a flood of repeat connections from re-reporting the same address before the bouncer picks it up. +/// +public sealed class BlocklistFilter : IConnectionFilter +{ + private static readonly ILogger logger = LogFactory.GetLogger(typeof(BlocklistFilter)); + + // Written by the reload poll (off-loop), read by the accept path (game loop): a single volatile + // reference swap is the whole synchronization story — readers see the old or the new snapshot, whole. + private volatile BlocklistSnapshot _snapshot = BlocklistSnapshot.Empty; + + private readonly PromotedGuard _guard = new(); + + private string _path; + private TimeSpan _interval; + private bool _reportHits; + private TimeSpan _banDuration; + private long _suppressionMs; + private string _lastGenerated; + private DateTime _lastWriteUtc; + private CancellationTokenSource _cts; + + public string Name => "blocklist"; + + public int Count => _snapshot.Count; + + public static void Configure() + { + ConnectionFilters.Register(new BlocklistFilter()); + } + + public void Register() + { + BlocklistConfiguration.Load(); + var s = BlocklistConfiguration.Settings; + + _path = ResolvePath(s.File); + _interval = s.ReloadInterval <= TimeSpan.Zero ? TimeSpan.FromSeconds(60) : s.ReloadInterval; + _reportHits = s.ReportHits; + _banDuration = s.BanDuration; + _suppressionMs = (long)s.PromoteSuppression.TotalMilliseconds; + } + + /// + /// Resolves the configured path once: a relative path is anchored to + /// (never the process working directory, which differs when the shard is launched from elsewhere), an + /// absolute path is taken as-is so several shards can share one generated list. + /// + private static string ResolvePath(string configured) + { + if (string.IsNullOrWhiteSpace(configured)) + { + return null; + } + + return Path.IsPathRooted(configured) ? configured : Path.Join(Core.BaseDirectory, configured); + } + + public void Start(CancellationToken token) + { + if (_path == null) + { + logger.Information("Blocklist disabled (\"file\" empty in blocklist.json)"); + return; + } + + _cts = CancellationTokenSource.CreateLinkedTokenSource(token); + + // A missing file is the shipped default, not an error: the gate stays inert until the poll picks + // up whatever the generator first writes. No restart needed. + if (File.Exists(_path)) + { + Reload(); // synchronous prime; empty on failure (fail-open) + } + else + { + logger.Information("Blocklist inert: no list at \"{Path}\"; polling every {Interval}", _path, _interval); + } + + // Sweep the promote-guard so a distinct-IP flood cannot grow it unbounded. + Timer.DelayCall(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1), SweepGuard); + + _ = Task.Run(() => PollLoop(_cts.Token), _cts.Token); + } + + public void Stop() + { + _cts?.Cancel(); + _cts?.Dispose(); + _cts = null; + } + + public bool ShouldDeny(IPAddress address) + { + if (!Evaluate(address, Core.TickCount, out var shouldReport)) + { + return false; + } + + if (shouldReport) + { + // Demand-page this address up to the OS-level bouncer. Enqueue-only; never blocks the loop. + BanChannel.Report(address, _banDuration, "blocklist"); + } + + return true; + } + + /// + /// The pure decision, split out so the accept-path policy can be tested without a clock or a ban + /// channel. is true at most once per suppression window. + /// + internal bool Evaluate(IPAddress address, long nowTicks, out bool shouldReport) + { + shouldReport = false; + + if (!_snapshot.IsBanned(address)) + { + return false; + } + + if (_reportHits) + { + shouldReport = _guard.TryMark(address.ToUInt128(), nowTicks, _suppressionMs); + } + + return true; + } + + private void SweepGuard() => _guard.Sweep(Core.TickCount); + + // Test hook: inject a snapshot and policy without file I/O. + internal void LoadForTesting(BlocklistSnapshot snapshot, bool reportHits = true, long suppressionMs = 60000) + { + _snapshot = snapshot; + _reportHits = reportHits; + _suppressionMs = suppressionMs; + } + + private async ValueTask PollLoop(CancellationToken token) + { + while (!token.IsCancellationRequested) + { + try + { + await Task.Delay(_interval, token); + } + catch (OperationCanceledException) + { + return; + } + + try + { + if (ChangedSinceLastLoad()) + { + // Parsing millions of lines competes with a save for CPU and page cache, so yield until + // the world is written out. See the threading policy in CLAUDE.md (rules #3 and #10). + while (World.Saving || World.WorldState == WorldState.PendingSave) + { + await Task.Delay(TimeSpan.FromSeconds(1), token); + } + + Reload(); + } + } + catch (OperationCanceledException) + { + return; + } + catch (Exception e) + { + logger.Warning(e, "Blocklist reload check failed; keeping last snapshot ({Count})", Count); + } + } + } + + private bool ChangedSinceLastLoad() + { + try + { + var info = new FileInfo(_path); + if (!info.Exists) + { + return false; + } + + if (info.LastWriteTimeUtc == _lastWriteUtc) + { + return false; // cheapest guard + } + } + catch + { + return false; + } + + return !BlocklistFile.TryReadHeader(_path, out var h) || h.Generated != _lastGenerated; + } + + private void Reload() + { + // Capture the mtime/header BEFORE Load() so the markers describe the version being parsed, not + // one the producer swapped in mid-parse. Stale markers only cost an extra reload next poll; + // capturing after could skip a version entirely. + var writeUtc = default(DateTime); + try + { + writeUtc = new FileInfo(_path).LastWriteTimeUtc; + } + catch + { + /* keep default */ + } + + BlocklistFile.TryReadHeader(_path, out var h); + + var next = BlocklistFile.Load(_path, out var parsed, out var skipped); + _snapshot = next; // single volatile swap; readers see old or new whole + _lastGenerated = h.Generated; + _lastWriteUtc = writeUtc; + + logger.Information("Blocklist loaded {Parsed} entr(ies) ({Count} ranges, {Skipped} skipped) gen={Gen}", + parsed, next.Count, skipped, h.Generated); + } +} diff --git a/Projects/UOContent/Misc/Blocklist/BlocklistSnapshot.cs b/Projects/UOContent/Misc/Blocklist/BlocklistSnapshot.cs new file mode 100644 index 000000000..52839512c --- /dev/null +++ b/Projects/UOContent/Misc/Blocklist/BlocklistSnapshot.cs @@ -0,0 +1,205 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: BlocklistSnapshot.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Buffers.Text; +using System.Net; +using System.Net.Sockets; +using System.Text; +using Server.Collections; + +namespace Server.Network.Bans; + +/// +/// Immutable dual-stack blocklist. Singles and CIDRs are folded into a single sorted, coalesced +/// interval index per family: IPv4 as ranges (lean for the millions-strong common +/// case), IPv6 as ranges (empty unless the feed carries v6). Immutable → lock-free reads. +/// +public sealed class BlocklistSnapshot +{ + public static readonly BlocklistSnapshot Empty = new(SortedRangeIndex.Empty, SortedRangeIndex.Empty); + + private readonly SortedRangeIndex _v4; + private readonly SortedRangeIndex _v6; + + public int Count => _v4.Count + _v6.Count; + + private BlocklistSnapshot(SortedRangeIndex v4, SortedRangeIndex v6) + { + _v4 = v4; + _v6 = v6; + } + + /// + /// Parses a blocklist directly from its UTF-8/ASCII file bytes — one line at a time, splitting on + /// '\n' with no per-line string allocation. IPv4 singles and CIDRs are parsed straight from the + /// byte span; IPv6 (the rare path) decodes the single address token and defers to the framework parser. + /// Malformed lines increment and never throw. Build-time intermediates use + /// the multithreaded pool because this runs off the game loop on the reload/bootstrap thread. + /// + public static BlocklistSnapshot Build(ReadOnlySpan data, out int parsed, out int skipped) + { + parsed = 0; + skipped = 0; + + // Only the two final index arrays (allocated inside SortedRangeIndex.Build) hit the heap; every + // build-time buffer here is a pooled ref list. mt: true is required — this runs off the game loop. + using var v4 = PooledRefList.Range>.Create(mt: true); + using var v6 = PooledRefList.Range>.Create(mt: true); + + var rest = data; + while (!rest.IsEmpty) + { + ReadOnlySpan line; + var nl = rest.IndexOf((byte)'\n'); + if (nl >= 0) + { + line = rest[..nl]; + rest = rest[(nl + 1)..]; + } + else + { + line = rest; + rest = default; + } + + line = line[Ascii.Trim(line)]; + if (line.IsEmpty || line[0] == (byte)'#' || line[0] == (byte)';') + { + continue; + } + + var slash = line.IndexOf((byte)'/'); + var addr = slash >= 0 ? line[..slash] : line; + var bitsToken = slash >= 0 ? line[(slash + 1)..] : default; + + if (addr.IndexOf((byte)':') < 0) + { + // IPv4 single or CIDR — parsed straight from the byte span. + if (slash >= 0) + { + if (IPAddressUtility.TryParseV4(addr, out var ip) && + TryParseBits(bitsToken, out var bits) && bits is >= 0 and <= 32) + { + var size = bits == 0 ? 0xFFFFFFFFu : (1u << (32 - bits)) - 1; + var b = ip & ~size; + v4.Add(new SortedRangeIndex.Range(b, b + size)); + parsed++; + } + else + { + skipped++; + } + } + else if (IPAddressUtility.TryParseV4(addr, out var ip)) + { + v4.Add(new SortedRangeIndex.Range(ip, ip)); + parsed++; + } + else + { + skipped++; + } + } + else if (TryDecodeV6(addr, out var v)) + { + // IPv6 is rare in these feeds; the single token was decoded and framework-parsed above. + if (slash >= 0) + { + if (TryParseBits(bitsToken, out var bits) && bits is >= 0 and <= 128) + { + var mask = bits == 0 ? UInt128.Zero : ~((UInt128.One << (128 - bits)) - 1); + var b = v & mask; + v6.Add(new SortedRangeIndex.Range(b, b | ~mask)); + parsed++; + } + else + { + skipped++; + } + } + else + { + v6.Add(new SortedRangeIndex.Range(v, v)); + parsed++; + } + } + else + { + skipped++; + } + } + + v4.Sort(SortedRangeIndex.ByMin); + v6.Sort(SortedRangeIndex.ByMin); + return new BlocklistSnapshot(SortedRangeIndex.Build(v4.AsSpan()), SortedRangeIndex.Build(v6.AsSpan())); + } + + // Decodes a single IPv6 address token from ASCII bytes and validates it via the framework parser. + private static bool TryDecodeV6(ReadOnlySpan addr, out UInt128 v) + { + v = UInt128.Zero; + if (addr.Length > 45) + { + return false; + } + + Span chars = stackalloc char[addr.Length]; + for (var i = 0; i < addr.Length; i++) + { + chars[i] = (char)addr[i]; + } + + if (!IPAddress.TryParse(chars, out var a) || a.AddressFamily != AddressFamily.InterNetworkV6) + { + return false; + } + + v = a.ToUInt128(); + return true; + } + + private static bool TryParseBits(ReadOnlySpan token, out int bits) + { + if (Utf8Parser.TryParse(token, out bits, out var consumed) && consumed == token.Length) + { + return true; + } + + bits = 0; + return false; + } + + public bool IsBanned(IPAddress ip) + { + if (ip.IsIPv4MappedToIPv6) + { + // v6-encoded v4 must not dodge the v4 set; extract the embedded v4 uint directly. + return IPAddressUtility.TryMappedV4(ip, out var mv) && _v4.Contains(mv); + } + + if (ip.AddressFamily == AddressFamily.InterNetwork) + { + return IPAddressUtility.TryV4(ip, out var v) && _v4.Contains(v); + } + + if (ip.AddressFamily == AddressFamily.InterNetworkV6) + { + return _v6.Contains(ip.ToUInt128()); + } + + return false; + } +} diff --git a/Projects/UOContent/Misc/Blocklist/PromotedGuard.cs b/Projects/UOContent/Misc/Blocklist/PromotedGuard.cs new file mode 100644 index 000000000..4f820b16e --- /dev/null +++ b/Projects/UOContent/Misc/Blocklist/PromotedGuard.cs @@ -0,0 +1,55 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: PromotedGuard.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; + +namespace Server.Network.Bans; + +/// Suppresses re-reporting the same IP within a TTL. Accept-path thread only. +public sealed class PromotedGuard +{ + private readonly Dictionary _expiry = []; + + public bool TryMark(UInt128 ip, long nowTicks, long ttlMs) + { + if (_expiry.TryGetValue(ip, out var exp) && exp - nowTicks > 0) + { + return false; + } + _expiry[ip] = nowTicks + ttlMs; + return true; + } + + public void Sweep(long nowTicks) + { + if (_expiry.Count == 0) + { + return; + } + using var dead = Collections.PooledRefQueue.Create(); + foreach (var (ip, exp) in _expiry) + { + if (exp - nowTicks <= 0) + { + dead.Enqueue(ip); + } + } + while (dead.Count > 0) + { + _expiry.Remove(dead.Dequeue()); + } + } +} diff --git a/Projects/UOContent/Misc/CrowdSec/CrowdSecAlert.cs b/Projects/UOContent/Misc/CrowdSec/CrowdSecAlert.cs new file mode 100644 index 000000000..db0d8d80f --- /dev/null +++ b/Projects/UOContent/Misc/CrowdSec/CrowdSecAlert.cs @@ -0,0 +1,65 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecAlert.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Text.Json.Serialization; + +namespace Server.Network.Bans.CrowdSec; + +/// One CrowdSec alert (POST /v1/alerts takes an array of these). +public sealed class CrowdSecAlert +{ + [JsonPropertyName("scenario")] public string Scenario { get; set; } + [JsonPropertyName("message")] public string Message { get; set; } + [JsonPropertyName("events_count")] public int EventsCount { get; set; } = 1; + [JsonPropertyName("start_at")] public string StartAt { get; set; } + [JsonPropertyName("stop_at")] public string StopAt { get; set; } + [JsonPropertyName("capacity")] public int Capacity { get; set; } + [JsonPropertyName("leakspeed")] public string LeakSpeed { get; set; } = "0s"; + [JsonPropertyName("simulated")] public bool Simulated { get; set; } + [JsonPropertyName("events")] public object[] Events { get; set; } = []; + [JsonPropertyName("remediation")] public bool Remediation { get; set; } = true; + [JsonPropertyName("source")] public CrowdSecSource Source { get; set; } + [JsonPropertyName("decisions")] public CrowdSecDecisionDto[] Decisions { get; set; } +} + +public sealed class CrowdSecSource +{ + [JsonPropertyName("scope")] public string Scope { get; set; } = "Ip"; + [JsonPropertyName("value")] public string Value { get; set; } +} + +public sealed class CrowdSecDecisionDto +{ + [JsonPropertyName("origin")] public string Origin { get; set; } + [JsonPropertyName("type")] public string Type { get; set; } = "ban"; + [JsonPropertyName("scope")] public string Scope { get; set; } = "Ip"; + [JsonPropertyName("value")] public string Value { get; set; } + [JsonPropertyName("duration")] public string Duration { get; set; } + [JsonPropertyName("scenario")] public string Scenario { get; set; } +} + +/// Watcher login request/response for POST /v1/watchers/login. +public sealed class CrowdSecLoginRequest +{ + [JsonPropertyName("machine_id")] public string MachineId { get; set; } + [JsonPropertyName("password")] public string Password { get; set; } +} + +public sealed class CrowdSecLoginResponse +{ + [JsonPropertyName("code")] public int Code { get; set; } + [JsonPropertyName("token")] public string Token { get; set; } + [JsonPropertyName("expire")] public string Expire { get; set; } +} diff --git a/Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs b/Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs new file mode 100644 index 000000000..af853244a --- /dev/null +++ b/Projects/UOContent/Misc/CrowdSec/CrowdSecAlertClient.cs @@ -0,0 +1,137 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecAlertClient.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Net.Http.Json; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; + +namespace Server.Network.Bans.CrowdSec; + +/// Reporter-side LAPI operations, mockable for tests. +public interface ICrowdSecAlertClient : IDisposable +{ + ValueTask PostAlertsAsync(IReadOnlyList alerts, CancellationToken token); + ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token); +} + +/// +/// CrowdSec LAPI watcher client: authenticates with machine credentials and posts/deletes decisions. +/// Holds a JWT refreshed on expiry or a 401. +/// +public sealed class CrowdSecAlertClient : ICrowdSecAlertClient +{ + private static readonly JsonSerializerOptions _jsonOptions = new() { PropertyNameCaseInsensitive = true }; + + private readonly HttpClient _http; + private readonly string _machineId; + private readonly string _password; + + private string _token; + private DateTime _tokenExpiresUtc = DateTime.MinValue; + + public CrowdSecAlertClient(CrowdSecSettings settings) + { + var baseUri = new Uri(settings.LapiUrl, UriKind.Absolute); // fails loud on malformed url + _http = new HttpClient { BaseAddress = baseUri, Timeout = TimeSpan.FromSeconds(30) }; + _http.DefaultRequestHeaders.Add("User-Agent", "ModernUO-watcher/1.0"); + _machineId = settings.MachineId; + _password = settings.Password; + } + + private async Task EnsureAuthAsync(CancellationToken token) + { + if (_token != null && DateTime.UtcNow < _tokenExpiresUtc - TimeSpan.FromMinutes(1)) + { + return; + } + + var request = new CrowdSecLoginRequest { MachineId = _machineId, Password = _password }; + using var response = await _http.PostAsJsonAsync("/v1/watchers/login", request, _jsonOptions, token) + .ConfigureAwait(false); + response.EnsureSuccessStatusCode(); + + var login = await response.Content.ReadFromJsonAsync(_jsonOptions, token) + .ConfigureAwait(false); + _token = login?.Token ?? throw new InvalidOperationException("CrowdSec login returned no token."); + _tokenExpiresUtc = DateTime.TryParse(login.Expire, out var exp) ? exp.ToUniversalTime() : DateTime.UtcNow.AddHours(1); + } + + private void Authorize(HttpRequestMessage message) => + message.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _token); + + public async ValueTask PostAlertsAsync(IReadOnlyList alerts, CancellationToken token) + { + if (alerts.Count == 0) + { + return; + } + + await SendWithRetryAsync(() => + { + var message = new HttpRequestMessage(HttpMethod.Post, "/v1/alerts") + { + Content = JsonContent.Create(alerts, options: _jsonOptions) + }; + Authorize(message); + return message; + }, token).ConfigureAwait(false); + } + + public async ValueTask DeleteDecisionsAsync(string origin, IPAddress ip, CancellationToken token) + { + var query = BuildDeleteQuery(origin, ip); + await SendWithRetryAsync(() => + { + var message = new HttpRequestMessage(HttpMethod.Delete, query); + Authorize(message); + return message; + }, token).ConfigureAwait(false); + } + + /// + /// Builds the decisions-delete query string. is operator-controlled config + /// (crowdsec.json), so it must be escaped like any other untrusted value going into a URL. + /// + internal static string BuildDeleteQuery(string origin, IPAddress ip) => + $"/v1/decisions?origin={Uri.EscapeDataString(origin)}&ip={Uri.EscapeDataString(ip.ToString())}"; + + private async ValueTask SendWithRetryAsync(Func build, CancellationToken token) + { + await EnsureAuthAsync(token).ConfigureAwait(false); + + using var first = build(); + using var response = await _http.SendAsync(first, token).ConfigureAwait(false); + if (response.StatusCode != HttpStatusCode.Unauthorized) + { + response.EnsureSuccessStatusCode(); + return; + } + + // Token rejected mid-flight: force a re-login and retry once. + _token = null; + await EnsureAuthAsync(token).ConfigureAwait(false); + using var retry = build(); + using var retryResponse = await _http.SendAsync(retry, token).ConfigureAwait(false); + retryResponse.EnsureSuccessStatusCode(); + } + + public void Dispose() => _http.Dispose(); +} diff --git a/Projects/UOContent/Misc/CrowdSec/CrowdSecConfiguration.cs b/Projects/UOContent/Misc/CrowdSec/CrowdSecConfiguration.cs new file mode 100644 index 000000000..b1376b6a0 --- /dev/null +++ b/Projects/UOContent/Misc/CrowdSec/CrowdSecConfiguration.cs @@ -0,0 +1,101 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecConfiguration.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.IO; +using System.Text.Json.Serialization; +using Server.Json; + +namespace Server.Network.Bans.CrowdSec; + +/// +/// Loads the from Configuration/crowdsec.json (matching the +/// per-feature JSON config pattern used by AssistantConfiguration). Loaded once; a missing file +/// writes a disabled-by-default template so operators have something to edit. +/// +public static class CrowdSecConfiguration +{ + private const string _path = "Configuration/crowdsec.json"; + + public static CrowdSecSettings Settings { get; private set; } + + public static void Load() + { + var path = Path.Join(Core.BaseDirectory, _path); + + if (File.Exists(path)) + { + Settings = JsonConfig.Deserialize(path); + } + else + { + Settings = new CrowdSecSettings + { + LapiUrl = "http://127.0.0.1:8080", + MachineId = "", + Password = "", + Origin = "modernuo", + ManualBanDuration = TimeSpan.FromHours(168), + FlushInterval = TimeSpan.FromSeconds(1), + MaxQueue = 10000 + }; + + Save(); + } + } + + private static void Save() + { + JsonConfig.Serialize(Path.Join(Core.BaseDirectory, _path), Settings); + } +} + +/// +/// Bound configuration for . Read once at Configure(). +/// The reporter is inert unless is true. +/// +public record CrowdSecSettings +{ + /// LAPI endpoint. Default http://127.0.0.1:8080. + [JsonPropertyName("lapiUrl")] + public string LapiUrl { get; set; } = "http://127.0.0.1:8080"; + + /// Watcher machine id from cscli machines add. Empty disables reporting. + [JsonPropertyName("machineId")] + public string MachineId { get; set; } = ""; + + /// Watcher password paired with . + [JsonPropertyName("password")] + public string Password { get; set; } = ""; + + /// Decision origin stamped on our contributions. Default modernuo. + [JsonPropertyName("origin")] + public string Origin { get; set; } = "modernuo"; + + /// Duration for manual admin bans pushed to CrowdSec. Default 168h (renewable). Finite so a missed retract self-heals. + [JsonPropertyName("manualBanDuration")] + public TimeSpan ManualBanDuration { get; set; } = TimeSpan.FromHours(168); + + /// Max time the drain coalesces before flushing a batch. Default 1s. + [JsonPropertyName("flushInterval")] + public TimeSpan FlushInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// Bounded contribution queue capacity; overflow is dropped (counted). Default 10000. + [JsonPropertyName("maxQueue")] + public int MaxQueue { get; set; } = 10000; + + [JsonIgnore] + public bool ReportingEnabled => !string.IsNullOrWhiteSpace(MachineId) && !string.IsNullOrWhiteSpace(Password); +} diff --git a/Projects/UOContent/Misc/CrowdSec/CrowdSecReporter.cs b/Projects/UOContent/Misc/CrowdSec/CrowdSecReporter.cs new file mode 100644 index 000000000..306d4ab53 --- /dev/null +++ b/Projects/UOContent/Misc/CrowdSec/CrowdSecReporter.cs @@ -0,0 +1,399 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: CrowdSecReporter.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; +using Server.Logging; + +namespace Server.Network.Bans.CrowdSec; + +/// +/// Contributes locally-decided bans to CrowdSec via the LAPI alerts API. Non-blocking on the accept +/// path: enqueues onto a bounded, drop-on-overflow channel drained by a single +/// background task that coalesces by IP and POSTs batched alerts. +/// +public sealed class CrowdSecReporter : IBanReporter +{ + private static readonly ILogger logger = LogFactory.GetLogger(typeof(CrowdSecReporter)); + + internal readonly record struct ReportItem(IPAddress Ip, TimeSpan Ttl, string Reason, bool Retract); + + // Bounded retry for transient LAPI failures during a drain send (network blips, 5xx). Distinct from + // CrowdSecAlertClient.SendWithRetryAsync's single 401-relogin retry, which is an auth concern. + private static readonly TimeSpan[] _retryDelays = [TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2)]; + + private ICrowdSecAlertClient _client; + private CrowdSecSettings _settings; + private Channel _queue; + private CancellationTokenSource _cts; + private Task _drainTask; + private int _dropped; + private int _sendFailures; + + public CrowdSecReporter() + { + } + + // Test/embedding ctor with an injected client + settings. + internal CrowdSecReporter(ICrowdSecAlertClient client, CrowdSecSettings settings) + { + _client = client; + _settings = settings; + _queue = CreateQueue(settings.MaxQueue); + } + + public string Name => "crowdsec"; + public bool CanRetract => true; + public int DroppedCount => _dropped; + + /// + /// Batches ultimately dropped after the bounded transient-retry in gave up. + /// Distinct from (queue-overflow drops on the accept path): this counts + /// sustained LAPI outages so operators can see contribution loss instead of it being silent. + /// + public int SendFailureCount => _sendFailures; + + /// The drain loop's task, so tests can assert it stays alive until the loop exits. + internal Task DrainTaskForTesting => _drainTask; + + public static void Configure() + { + BanChannel.Register(new CrowdSecReporter()); + } + + public void Register() + { + CrowdSecConfiguration.Load(); + _settings ??= CrowdSecConfiguration.Settings; + } + + public void Start(CancellationToken token) + { + if (!_settings.ReportingEnabled) + { + logger.Information("CrowdSec reporter disabled (machineId/password empty in crowdsec.json)"); + return; + } + + _client ??= new CrowdSecAlertClient(_settings); + _queue ??= CreateQueue(_settings.MaxQueue); + _cts = CancellationTokenSource.CreateLinkedTokenSource(token); + _drainTask = Task.Run(() => DrainLoop(_cts.Token), _cts.Token); + } + + public void Stop() + { + _cts?.Cancel(); + + // The flush below reads a SingleReader channel, so wait for the drain to actually exit first. + var drainExited = true; + try + { + // Wait(timeout) is false only on timeout; a throw means faulted/cancelled, which is still exited. + drainExited = _drainTask == null || _drainTask.Wait(TimeSpan.FromSeconds(2)); + } + catch + { + // Ignored: a faulted wait means the drain has completed and released the channel. + } + + _cts?.Dispose(); + _cts = null; + + if (drainExited) + { + FlushRemainingOnStop(); + } + + _client?.Dispose(); + _drainTask = null; + } + + /// + /// Best-effort bounded flush of whatever is still queued at shutdown. Blocking is correct here — the + /// loop has stopped ticking — but must not happen on the loop thread: runs where + /// SynchronizationContext.Current is the EventLoopContext, and a captured continuation + /// would be posted to a queue nothing pumps any more. keeps the + /// chain on the pool; the bounded wait caps a wedged send at a few seconds of shutdown. + /// + private void FlushRemainingOnStop() + { + if (_queue == null || _client == null) + { + return; + } + + _queue.Writer.TryComplete(); + + List reports = []; + List retracts = []; + while (_queue.Reader.TryRead(out var item)) + { + (item.Retract ? retracts : reports).Add(item); + } + + if (reports.Count == 0 && retracts.Count == 0) + { + return; + } + + try + { + if (!Task.Run(() => FlushRemainingOnStopAsync(reports, retracts)).Wait(TimeSpan.FromSeconds(4))) + { + logger.Warning( + "CrowdSec flush-on-stop timed out; {Count} item(s) not contributed", + reports.Count + retracts.Count + ); + } + } + catch (Exception e) + { + logger.Warning(e, "CrowdSec flush-on-stop failed"); + } + } + + /// + /// Uses a fresh token, not the drain loop's already-cancelled one, which would fail every send + /// immediately. Reports go as one deduped batch; retracts go as individual DELETEs so an admin's + /// unban propagates on a clean shutdown. Leftovers self-heal via + /// . + /// + private async Task FlushRemainingOnStopAsync(List reports, List retracts) + { + using var flushCts = new CancellationTokenSource(TimeSpan.FromSeconds(3)); + + if (reports.Count > 0) + { + var alerts = BuildAlerts(reports, _settings, DateTime.UtcNow); + try + { + await _client.PostAlertsAsync(alerts, flushCts.Token).ConfigureAwait(false); + } + catch (Exception e) + { + logger.Warning(e, "CrowdSec flush-on-stop reports failed"); + RecordSendFailure(alerts.Count); + } + } + + HashSet seen = []; + for (var i = 0; i < retracts.Count; i++) + { + if (flushCts.IsCancellationRequested) + { + break; // out of budget; the rest self-heal via ManualBanDuration + } + + var ip = retracts[i].Ip; + if (!seen.Add(ip.ToString())) + { + continue; + } + + try + { + await _client.DeleteDecisionsAsync(_settings.Origin, ip, flushCts.Token).ConfigureAwait(false); + } + catch (Exception e) + { + logger.Warning(e, "CrowdSec flush-on-stop retract failed for {Address}", ip); + RecordSendFailure(1); + } + } + } + + public void Report(IPAddress address, TimeSpan ttl, string reason) => + Enqueue(new ReportItem(address, ttl, reason, false)); + + public void Retract(IPAddress address) => + Enqueue(new ReportItem(address, TimeSpan.Zero, "retract", true)); + + private void Enqueue(ReportItem item) + { + if (_queue == null || !_queue.Writer.TryWrite(item)) + { + Interlocked.Increment(ref _dropped); + } + } + + // FullMode.Wait (the default) makes TryWrite return false immediately when the channel is full + // instead of blocking the caller — exactly the non-blocking drop-on-overflow behavior the accept + // path requires. DropWrite would silently discard the new item and always report success, which + // would make overflow undetectable. + private static Channel CreateQueue(int capacity) => + Channel.CreateBounded(new BoundedChannelOptions(Math.Max(1, capacity)) + { + FullMode = BoundedChannelFullMode.Wait, + SingleReader = true + }); + + // Must return Task: Start() passes this to Task.Run, which has no Func overload, so a + // ValueTask would bind to Task.Run and yield a Task that completes at the first + // await rather than when the loop exits. + private async Task DrainLoop(CancellationToken token) + { + var reader = _queue.Reader; + + while (!token.IsCancellationRequested) + { + try + { + if (!await reader.WaitToReadAsync(token).ConfigureAwait(false)) + { + return; + } + + // Coalesce a burst before flushing. + await Task.Delay(_settings.FlushInterval, token).ConfigureAwait(false); + + List reports = []; + List retracts = []; + while (reader.TryRead(out var item)) + { + (item.Retract ? retracts : reports).Add(item); + } + + if (reports.Count > 0) + { + var alerts = BuildAlerts(reports, _settings, DateTime.UtcNow); + if (!await SendWithBoundedRetryAsync(() => _client.PostAlertsAsync(alerts, token), token) + .ConfigureAwait(false)) + { + RecordSendFailure(alerts.Count); + } + } + + for (var i = 0; i < retracts.Count; i++) + { + var ip = retracts[i].Ip; + if (!await SendWithBoundedRetryAsync(() => _client.DeleteDecisionsAsync(_settings.Origin, ip, token), token) + .ConfigureAwait(false)) + { + RecordSendFailure(1); + } + } + } + catch (OperationCanceledException) + { + return; + } + catch (Exception e) + { + // Contribution is auxiliary: log and keep draining. Never crash the shard. + logger.Warning(e, "CrowdSec contribution flush failed; dropped this batch"); + } + } + } + + /// + /// Sends with up to 3 attempts total (1 initial + 2 retries), backing off 1s then 2s between + /// attempts, for transient LAPI failures (network blips, 5xx). Backoff uses + /// so it never blocks the thread; a cancellation during backoff propagates as + /// so the drain loop exits cleanly. Returns false (never + /// throws for a send failure) once attempts are exhausted, so the caller can count the drop and keep + /// draining instead of losing the rest of the batch/queue. + /// + private static async ValueTask SendWithBoundedRetryAsync(Func send, CancellationToken token) + { + for (var attempt = 0; ; attempt++) + { + try + { + await send().ConfigureAwait(false); + return true; + } + catch (OperationCanceledException) + { + throw; + } + catch (Exception e) + { + if (attempt >= _retryDelays.Length) + { + logger.Warning(e, "CrowdSec send failed after {Attempts} attempt(s); giving up", attempt + 1); + return false; + } + + var delay = _retryDelays[attempt]; + logger.Warning(e, "CrowdSec send failed (attempt {Attempt}); retrying in {Delay}", attempt + 1, delay); + await Task.Delay(delay, token).ConfigureAwait(false); + } + } + } + + private void RecordSendFailure(int itemCount) + { + var total = Interlocked.Increment(ref _sendFailures); + logger.Warning( + "CrowdSec contribution batch dropped after retries ({Items} item(s)); total dropped batches: {Total}", + itemCount, + total + ); + } + + /// Coalesces items by IP (last write wins) and builds one alert per unique address. + internal static List BuildAlerts(IEnumerable items, CrowdSecSettings settings, DateTime nowUtc) + { + Dictionary byIp = []; + foreach (var item in items) + { + byIp[item.Ip.ToString()] = item; + } + + var timestamp = nowUtc.ToString("yyyy-MM-ddTHH:mm:ss.fffZ"); + var alerts = new List(byIp.Count); + + foreach (var (value, item) in byIp) + { + var ttl = item.Reason == "manual" || item.Ttl <= TimeSpan.Zero ? settings.ManualBanDuration : item.Ttl; + var scenario = $"{settings.Origin}/{item.Reason}"; + + alerts.Add(new CrowdSecAlert + { + Scenario = scenario, + Message = $"ModernUO {item.Reason} ban for {value}", + StartAt = timestamp, + StopAt = timestamp, + Source = new CrowdSecSource { Scope = "Ip", Value = value }, + Decisions = + [ + new CrowdSecDecisionDto + { + Origin = settings.Origin, + Type = "ban", + Scope = "Ip", + Value = value, + Duration = FormatDuration(ttl), + Scenario = scenario + } + ] + }); + } + + return alerts; + } + + /// CrowdSec accepts Go durations; whole seconds are unambiguous and sufficient. + internal static string FormatDuration(TimeSpan ttl) + { + var seconds = (long)ttl.TotalSeconds; + return $"{Math.Max(1, seconds)}s"; + } +} diff --git a/Projects/Server/Network/Firewall/BaseFirewallEntry.cs b/Projects/UOContent/Misc/Firewall/BaseFirewallEntry.cs similarity index 100% rename from Projects/Server/Network/Firewall/BaseFirewallEntry.cs rename to Projects/UOContent/Misc/Firewall/BaseFirewallEntry.cs diff --git a/Projects/Server/Network/Firewall/CidrFirewallEntry.cs b/Projects/UOContent/Misc/Firewall/CidrFirewallEntry.cs similarity index 70% rename from Projects/Server/Network/Firewall/CidrFirewallEntry.cs rename to Projects/UOContent/Misc/Firewall/CidrFirewallEntry.cs index db74cf620..06527ab71 100644 --- a/Projects/Server/Network/Firewall/CidrFirewallEntry.cs +++ b/Projects/UOContent/Misc/Firewall/CidrFirewallEntry.cs @@ -25,8 +25,15 @@ public class CidrFirewallEntry : BaseFirewallEntry public override UInt128 MaxIpAddress { get; } public CidrFirewallEntry(string ipAddressOrCidr) - : this(ParseIPAddress(ipAddressOrCidr, out var prefixLength), prefixLength) { + // Core owns the CIDR -> normalized range parse. + if (!IPAddressUtility.TryParseCidrRange(ipAddressOrCidr, out var min, out var max)) + { + throw new ArgumentException("Invalid IP address or CIDR.", nameof(ipAddressOrCidr)); + } + + MinIpAddress = min; + MaxIpAddress = max; } public CidrFirewallEntry(IPAddress minAddress, IPAddress maxAddress) @@ -50,26 +57,4 @@ public class CidrFirewallEntry : BaseFirewallEntry MaxIpAddress = Utility.CreateCidrAddress(bytes, prefixLength, true); } - private static IPAddress ParseIPAddress(ReadOnlySpan ipString, out int prefixLength) - { - var slashIndex = ipString.IndexOf('/'); - var ipAddress = IPAddress.Parse(slashIndex > -1 ? ipString[..slashIndex] : ipString); - var maxPrefixLength = ipAddress.AddressFamily == AddressFamily.InterNetworkV6 ? 128 : 32; - - if (slashIndex == -1) - { - prefixLength = maxPrefixLength; - } - else - { - var prefixPart = ipString[(slashIndex + 1)..]; - - if (!int.TryParse(prefixPart, out prefixLength) || prefixLength < 0 || prefixLength > maxPrefixLength) - { - throw new ArgumentException("Invalid prefix length."); - } - } - - return ipAddress; - } } diff --git a/Projects/UOContent/Misc/Firewall/Firewall.cs b/Projects/UOContent/Misc/Firewall/Firewall.cs new file mode 100644 index 000000000..075b47ec3 --- /dev/null +++ b/Projects/UOContent/Misc/Firewall/Firewall.cs @@ -0,0 +1,413 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: Firewall.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Net; +using System.Runtime.CompilerServices; +using Server.Collections; +using Server.Json; +using Server.Logging; + +namespace Server.Network; + +public static class Firewall +{ + // Single-threaded: the accept path, admin gump/command, TTL expiry timer, and boot load all run on + // the main game loop. No locks, caches, or version counters are needed. See the ban-channel design doc. + // _entries is the authoritative store (gump/persistence/TTL/command all work against it); _index is a + // derived, rebuild-on-demand SortedRangeIndex used only for the accept-path IsBlocked lookup, shared + // with the same sorted-range binary-search primitive the blocklist uses (see BlocklistSnapshot). + private static readonly List _entries = []; + + // Entries with a TTL: entry -> absolute expiry tick (Core.TickCount). Permanent entries are absent. + private static readonly Dictionary _expiring = []; + + private static SortedRangeIndex _index = SortedRangeIndex.Empty; + private static bool _indexDirty; + + private static readonly ILogger logger = LogFactory.GetLogger(typeof(Firewall)); + private const string _path = "Configuration/firewall.json"; + private const string _legacyPath = "firewall.cfg"; + private static bool _dirty; + private static bool _configured; + + public static int FirewallSetCount => _entries.Count; + + public static void ReadFirewallSet(Action> callback) => callback(_entries); + + public static bool IsBlocked(IPAddress address) + { + if (_entries.Count == 0) + { + return false; + } + + EnsureIndex(); + return _index.Contains(address.ToUInt128()); + } + + // Rebuilds the derived lookup index from the authoritative _entries list, but only when entries have + // changed since the last build. Runs on the main game loop, so the pooled build buffer is single-threaded + // (mt: false); only the two final SortedRangeIndex arrays are heap-allocated. + private static void EnsureIndex() + { + if (!_indexDirty) + { + return; + } + + using var ranges = PooledRefList.Range>.Create(_entries.Count, mt: false); + for (var i = 0; i < _entries.Count; i++) + { + var entry = _entries[i]; + ranges.Add(new SortedRangeIndex.Range(entry.MinIpAddress, entry.MaxIpAddress)); + } + + ranges.Sort(SortedRangeIndex.ByMin); + _index = SortedRangeIndex.Build(ranges.AsSpan()); + _indexDirty = false; + } + + public static bool Add(IFirewallEntry firewallEntry) => Add(firewallEntry, TimeSpan.Zero); + + /// + /// Adds an entry. <= means permanent. Returns false + /// if the entry was already present. + /// + // Indexed scan (no closure allocation); firewall lists are small, so O(n) is negligible and this + // stays off the hot path (Add/Remove are admin/boot actions, not the accept path). + private static int IndexOfEntry(IFirewallEntry entry) + { + for (var i = 0; i < _entries.Count; i++) + { + if (_entries[i].CompareTo(entry) == 0) + { + return i; + } + } + + return -1; + } + + public static bool Add(IFirewallEntry firewallEntry, TimeSpan ttl, bool persist = true) + { + if (firewallEntry == null || IndexOfEntry(firewallEntry) >= 0) + { + return false; + } + + _entries.Add(firewallEntry); + + if (ttl > TimeSpan.Zero) + { + _expiring[firewallEntry] = Core.TickCount + (long)ttl.TotalMilliseconds; + } + + _indexDirty = true; + + if (persist) + { + MarkDirty(); + } + + return true; + } + + public static bool Remove(IFirewallEntry entry) + { + if (entry == null) + { + return false; + } + + var index = IndexOfEntry(entry); + if (index < 0) + { + return false; + } + + // Remove the stored instance from _expiring (not the passed reference), so a value-equal + // entry created elsewhere still clears the TTL bookkeeping. + var stored = _entries[index]; + _entries.RemoveAt(index); + _expiring.Remove(stored); + _indexDirty = true; + MarkDirty(); + return true; + } + + /// + /// Removes every entry whose TTL has elapsed. Called from the main-thread maintenance timer (Task 2). + /// + internal static void ExpireEntries(long nowTicks) + { + if (_expiring.Count == 0) + { + return; + } + + List expired = null; + foreach (var (entry, expiresAt) in _expiring) + { + if (expiresAt - nowTicks <= 0) + { + (expired ??= []).Add(entry); + } + } + + if (expired == null) + { + return; + } + + for (var i = 0; i < expired.Count; i++) + { + var entry = expired[i]; + _entries.Remove(entry); + _expiring.Remove(entry); + } + + _indexDirty = true; + MarkDirty(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static IFirewallEntry ToFirewallEntry(object entry) => + entry switch + { + IFirewallEntry firewallEntry => firewallEntry, + IPAddress address => new SingleIpFirewallEntry(address), + string s => ToFirewallEntry(s), + _ => null + }; + + public static IFirewallEntry ToFirewallEntry(string entry) + { + if (entry == null) + { + return null; + } + + try + { + var rangeSeparator = entry.IndexOf('-'); + if (rangeSeparator > -1) + { + return new CidrFirewallEntry( + IPAddress.Parse(entry.AsSpan(0, rangeSeparator)), + IPAddress.Parse(entry.AsSpan(rangeSeparator + 1)) + ); + } + + if (entry.IndexOf('/') > -1) + { + return new CidrFirewallEntry(entry); + } + + return new SingleIpFirewallEntry(entry); + } + catch + { + return null; + } + } + + public static void Configure() + { + if (_configured) + { + return; + } + _configured = true; + + var path = Path.Join(Core.BaseDirectory, _path); + + if (File.Exists(path)) + { + LoadFrom(JsonConfig.Deserialize(path)); + } + else + { + var legacyPath = ResolveLegacyCfgPath(); + if (legacyPath != null) + { + MigrateLegacyCfg(legacyPath); + Save(); // materialize firewall.json; the .cfg is no longer read after this + TryMarkLegacyCfgMigrated(legacyPath); + } + } + + // Main-thread maintenance: expire TTLs and flush pending writes. No background thread. + Timer.DelayCall(TimeSpan.FromSeconds(30), TimeSpan.FromSeconds(30), Maintenance); + + // Expose the set to the accept path. Everything else (gump, commands, persistence) keeps using + // the Firewall API directly; only the per-connection question goes through the filter registry. + ConnectionFilters.Register(FirewallConnectionFilter.Instance); + } + + private static void Maintenance() + { + ExpireEntries(Core.TickCount); + + if (_dirty) + { + Save(); + } + } + + private static void MarkDirty() => _dirty = true; + + internal static void LoadFrom(FirewallSettings settings) + { + if (settings?.Entries == null) + { + return; + } + + // Core.Now: this runs on the game loop, via the Configure sweep. + var now = Core.Now; + var records = settings.Entries; + for (var i = 0; i < records.Length; i++) + { + var record = records[i]; + var entry = ToFirewallEntry(record.Value); + if (entry == null) + { + logger.Warning("Ignoring unparseable firewall entry \"{Entry}\"", record.Value); + continue; + } + + var ttl = TimeSpan.Zero; + if (record.Expires is { } expires) + { + ttl = expires - now; + if (ttl <= TimeSpan.Zero) + { + continue; // already expired + } + } + + Add(entry, ttl, persist: false); + } + } + + internal static FirewallSettings ToSettings() + { + // expires is derived below as now + (expiresAtTick - nowTicks), so both operands must come from + // the same instant. Core.Now and Core.TickCount are refreshed together each loop iteration; a + // fresh DateTime.UtcNow here would bake the loop's lag into every persisted expiry. + var now = Core.Now; + var nowTicks = Core.TickCount; + var list = new List(_entries.Count); + + for (var i = 0; i < _entries.Count; i++) + { + var entry = _entries[i]; + DateTime? expires = null; + if (_expiring.TryGetValue(entry, out var expiresAtTick)) + { + expires = now.AddMilliseconds(expiresAtTick - nowTicks); + } + + list.Add(new FirewallEntryRecord { Value = entry.ToString(), Expires = expires }); + } + + return new FirewallSettings { Entries = list.ToArray() }; + } + + public static void Save() + { + _dirty = false; + var path = Path.Join(Core.BaseDirectory, _path); + var tmp = $"{path}.tmp"; + JsonConfig.Serialize(tmp, ToSettings()); + File.Move(tmp, path, overwrite: true); // atomic swap + } + + /// + /// Locates the legacy firewall.cfg to migrate. The modern convention is , + /// checked first; the pre-collapse AdminFirewall used a bare relative path (resolved against the + /// process's current working directory), which may differ from when the + /// shard is launched from elsewhere, so that's checked as a fallback. Returns null if neither exists. + /// + private static string ResolveLegacyCfgPath() + { + var underBaseDirectory = Path.Join(Core.BaseDirectory, _legacyPath); + if (File.Exists(underBaseDirectory)) + { + return underBaseDirectory; + } + + return File.Exists(_legacyPath) ? _legacyPath : null; + } + + private static void MigrateLegacyCfg(string legacyPath) + { + var searchValues = SearchValues.Create("*Xx?"); + + using var reader = new StreamReader(legacyPath); + while (reader.ReadLine() is { } line) + { + line = line.Trim(); + if (line.Length == 0) + { + continue; + } + + if (line.AsSpan().ContainsAny(searchValues)) + { + logger.Warning("Legacy firewall entry \"{Entry}\" ignored during migration", line); + continue; + } + + var entry = ToFirewallEntry(line); + if (entry != null) + { + Add(entry, TimeSpan.Zero, persist: false); + } + } + + logger.Information("Migrated {Count} entr(ies) from legacy firewall.cfg to firewall.json", _entries.Count); + } + + /// + /// Renames the migrated .cfg to firewall.cfg.migrated so it isn't re-scanned on the next + /// boot and operators can see it was already migrated. Best-effort: a locked/read-only file must not + /// fail startup, since the migration itself (firewall.json) already succeeded. + /// + private static void TryMarkLegacyCfgMigrated(string legacyPath) + { + try + { + File.Move(legacyPath, $"{legacyPath}.migrated", overwrite: true); + } + catch (Exception e) + { + logger.Warning(e, "Could not rename migrated legacy firewall file \"{Path}\"", legacyPath); + } + } + + internal static void ResetForTesting() + { + _entries.Clear(); + _expiring.Clear(); + _index = SortedRangeIndex.Empty; + _indexDirty = false; + _configured = false; + } +} diff --git a/Projects/UOContent/Misc/Firewall/FirewallConnectionFilter.cs b/Projects/UOContent/Misc/Firewall/FirewallConnectionFilter.cs new file mode 100644 index 000000000..570e33f19 --- /dev/null +++ b/Projects/UOContent/Misc/Firewall/FirewallConnectionFilter.cs @@ -0,0 +1,52 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: FirewallConnectionFilter.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System.Net; +using System.Threading; + +namespace Server.Network; + +/// +/// Exposes the admin-curated set to the accept path as an +/// . The firewall keeps its own API (the admin gump and commands mutate +/// it directly); this is only the accept-path adapter, since a static class cannot implement an +/// interface. It is the cheapest gate to consult — an empty set costs one length compare — so +/// registers it first. +/// +internal sealed class FirewallConnectionFilter : IConnectionFilter +{ + public static readonly FirewallConnectionFilter Instance = new(); + + private FirewallConnectionFilter() + { + } + + public string Name => "firewall"; + + // Firewall.Configure() owns loading and registration. + public void Register() + { + } + + // Nothing to hydrate: the set loads at Configure and is maintained by a main-loop timer. + public void Start(CancellationToken token) + { + } + + /// Flushes pending writes on the way down so a TTL expiry or late admin edit is not lost. + public void Stop() => Firewall.Save(); + + public bool ShouldDeny(IPAddress address) => Firewall.IsBlocked(address); +} diff --git a/Projects/UOContent/Misc/Firewall/FirewallSettings.cs b/Projects/UOContent/Misc/Firewall/FirewallSettings.cs new file mode 100644 index 000000000..dd0281c92 --- /dev/null +++ b/Projects/UOContent/Misc/Firewall/FirewallSettings.cs @@ -0,0 +1,42 @@ +/************************************************************************* + * ModernUO * + * Copyright 2019-2026 - ModernUO Development Team * + * Email: hi@modernuo.com * + * File: FirewallSettings.cs * + * * + * This program is free software: you can redistribute it and/or modify * + * it under the terms of the GNU General Public License as published by * + * the Free Software Foundation, either version 3 of the License, or * + * (at your option) any later version. * + * * + * You should have received a copy of the GNU General Public License * + * along with this program. If not, see . * + *************************************************************************/ + +using System; +using System.Text.Json.Serialization; + +namespace Server.Network; + +/// +/// Persisted local firewall entries (manual admin bans). Stored at Configuration/firewall.json. +/// Auto-detected rate-limit trips are never persisted — they are contributed to CrowdSec, not stored here. +/// +public record FirewallSettings +{ + [JsonPropertyName("entries")] + public FirewallEntryRecord[] Entries { get; set; } = []; +} + +/// +/// One persisted entry. is a single IP, a min-max range, or CIDR. +/// is UTC wall-clock; null means permanent. +/// +public record FirewallEntryRecord +{ + [JsonPropertyName("value")] + public string Value { get; set; } + + [JsonPropertyName("expires")] + public DateTime? Expires { get; set; } +} diff --git a/Projects/Server/Network/Firewall/IFirewallEntry.cs b/Projects/UOContent/Misc/Firewall/IFirewallEntry.cs similarity index 100% rename from Projects/Server/Network/Firewall/IFirewallEntry.cs rename to Projects/UOContent/Misc/Firewall/IFirewallEntry.cs diff --git a/Projects/Server/Network/Firewall/SingleIpFirewallEntry.cs b/Projects/UOContent/Misc/Firewall/SingleIpFirewallEntry.cs similarity index 100% rename from Projects/Server/Network/Firewall/SingleIpFirewallEntry.cs rename to Projects/UOContent/Misc/Firewall/SingleIpFirewallEntry.cs diff --git a/dev-docs/networking-packets.md b/dev-docs/networking-packets.md index 43d4aec15..5eed62249 100644 --- a/dev-docs/networking-packets.md +++ b/dev-docs/networking-packets.md @@ -473,6 +473,65 @@ ns.SendMovementRej(int sequence, Mobile m); 6. **Big-endian by default** -- only use `WriteLE`/`ReadLE` when the protocol requires it 7. **Function pointers** (`&Handler`) for incoming packet registration (no delegate allocation) +## Connection Filtering (Accept Path) + +Every inbound socket is checked before it becomes a `NetState`. The check runs on the game loop once +per accepted connection -- this is the path that has to survive a DDoS -- so it must be allocation-free +and non-blocking. + +Gates plug in through `IConnectionFilter`, registered with `ConnectionFilters.Register()` during the +Configure sweep: + +```csharp +public sealed class MyFilter : IConnectionFilter +{ + public string Name => "my-filter"; + public void Configure() { /* read config, no I/O */ } + public void Start(CancellationToken token) { /* background hydration */ } + public void Stop() { } + public bool ShouldDeny(IPAddress address) => /* allocation-free membership test */; +} + +// In a static Configure() so the sweep finds it: +ConnectionFilters.Register(new MyFilter()); +``` + +Rules: + +- `ShouldDeny` must be **allocation-free**, O(log n) at worst, no I/O, no blocking. Anything expensive + (parsing, reloading, contributing to an external service) belongs off the loop or behind a bounded, + non-blocking enqueue. +- Side effects a hit implies (reporting to `BanChannel`, promoting to an OS firewall, suppressing + duplicate reports) are the **filter's** business, not the accept path's. +- Filters are consulted in registration order and the first denial short-circuits, so register the + cheapest and most selective first. Order affects only how quickly a denial is reached, never whether + one happens. +- A filter that throws is **unregistered** and the connection fails open. A filter that faults once + faults for every connection, so leaving it registered would mean an exception per accept. + +Core owns the question; **every implementation lives in UOContent**. The two that ship are `firewall` +(admin-curated, mutable at runtime, persisted to `Configuration/firewall.json`) and `blocklist` +(file-sourced, millions of entries, demand-pages hits to CrowdSec). A shard that fronts its server with +an upstream proxy or edge scrubbing can drop both and register nothing. + +Do **not** route this kind of check through `EventSink.InvokeSocketConnect` -- that fires later and +allocates a `SocketConnectEventArgs` per connection, which is exactly what the accept path avoids for +rejected traffic. + +### IP Address Normalization (`IPAddressUtility`) + +Addresses are normalized to `UInt128` in **IPv6 form** so a single comparison/index works for both +families. An IPv4 address becomes its v4-mapped-v6 value (`::ffff:a.b.c.d`), which is why round-tripping +matters: `IPv4 -> UInt128 -> IPAddress` can come back as `InterNetworkV6` with `IsIPv4MappedToIPv6` +set, even though it is "really" a v4 address. Code that switches on `AddressFamily` alone will mis-handle +those, so the helpers check both. + +> **Known wart / follow-up:** `ToUInt128` guards with `AddressFamily == InterNetwork && !IsIPv4MappedToIPv6`. +> Per BCL semantics `IsIPv4MappedToIPv6` is only ever true for `InterNetworkV6`, so the second clause +> reads as redundant -- it is really defending the round-trip described above. The normalization would be +> clearer as an explicit "to canonical v6 bits" step that never needs the family check at all. Deliberately +> left as-is; to be revisited in a follow-up PR rather than churned mid-feature. + ## Key File References | File | Description | @@ -492,3 +551,8 @@ ns.SendMovementRej(int sequence, Mobile m); | `Projects/Server/Network/Packets/OutgoingAccountPackets.cs` | Account packets | | `Projects/Server/Network/Packets/OutgoingContainerPackets.cs` | Container packets | | `Projects/Server/Network/PacketHandler.cs` | PacketHandler class | +| `Projects/Server/Network/IConnectionFilter.cs` | Accept-path gate contract | +| `Projects/Server/Network/ConnectionFilters.cs` | Filter registry + lifecycle | +| `Projects/UOContent/Misc/Firewall/Firewall.cs` | Admin-curated firewall set | +| `Projects/Server/Utilities/IPAddressUtility.cs` | IPAddress <-> UInt128 normalization, CIDR parsing | +| `Projects/UOContent/Misc/Blocklist/BlocklistFilter.cs` | File-sourced blocklist filter | diff --git a/tools/Export-IpBlocklist.ps1 b/tools/Export-IpBlocklist.ps1 new file mode 100644 index 000000000..81b9afd06 --- /dev/null +++ b/tools/Export-IpBlocklist.ps1 @@ -0,0 +1,543 @@ +#requires -Version 7.0 +<# +.SYNOPSIS + Downloads a small, non-overlapping set of public IP threat feeds and writes them to a single + ModernUO blocklist file -- merged, de-duplicated and bogon-filtered. + +.DESCRIPTION + This is the producer half of ModernUO's in-app blocklist gate. It fetches a deliberately THIN feed + set, merges every source into one global set, drops duplicates and reserved/bogon addresses, then + writes the result to a plain text file that the shard reads via `file` in + Configuration/blocklist.json. Nothing is installed and no credentials are needed -- the output is just + a text file, so this can run on any machine that can reach the shard's Distribution folder. + + It writes the file the shard's `BlocklistFilter` demand-pages against. IPs that actually connect are + promoted to CrowdSec / the OS firewall by the shard; the OS firewall never has to hold millions of + entries, which is exactly the scale it cannot handle on Windows. + + Inclusion principle: any category of IP used in OTHER attacks that could plausibly be turned against a + game server should be blocked -- compromised hosts, botnets, scanners, spam / DDoS-as-a-service bots, + open proxies and Tor relays. That whole surface is already covered by the anchor feed `bitwire-it`, + which is itself a 91-source aggregator (it folds in spamhaus, ipsum, firehol-level2, blocklist-de, + dshield, emergingthreats, binarydefense, cins-army, bruteforceblocker, greensnow, vxvault, ThreatFox, + StopForumSpam/sblam, Tor, open-proxy and C2 lists). So the inclusive posture lives in the base layer, + and every one of those standalone feeds is dropped as pure redundancy. Only the feeds bitwire does NOT + already carry are kept on top of it: + + bitwire-it 2h-refreshed 91-source aggregate (compromised hosts, botnets, scanners, spam + bots, Tor/open-proxy abuse relays, ThreatFox C2) -- the broad base layer. + romainmarcoux ~130k fresh attacker IPs bitwire's snapshot lags on (high-churn feed). + sentinel-turris ~800 unique honeypot probers (Turris greylist) not in bitwire. + firehol-level1 hijacked/reputation NETBLOCKS (spamhaus DROP-style) -- bogon-filtered. + + The only category deliberately held back is commercial VPN exit endpoints, which could block a legit + player -- and those are barely present here anyway (bitwire is ~5% of VPN-tunnel lists). If you ever want + to protect VPN/Tor players, pass -ExcludeAnonymizers to subtract Tor/open-proxy/VPN IPs from the output. + + OUTPUT FORMAT (must stay in sync with UOContent/Misc/Blocklist/BlocklistFile.cs): + Line 1 is a header comment carrying the version markers, e.g. + # modernuo-blocklist generated=2026-07-25T18:03:11Z count=3914022 ipv4=3901188 cidr=12834 + The shard polls `reloadInterval` and reloads when the file mtime AND `generated=` change, + so the header is REQUIRED -- without it the shard loads once and never picks up a new file. + Every following line is one entry: a bare IPv4/IPv6 address or a CIDR (`1.2.3.0/24`). Blank lines + and lines starting with `#` or `;` are ignored. Order does not matter; the shard sorts and + coalesces on load. The feeds used here are IPv4-only, but the shard parses IPv6 lines too. + + The file is written to a `.tmp` sibling and swapped into place atomically, so the shard never reads a + half-written list -- it either sees the previous version or the new one, whole. + + Performance: bitwire alone is ~4M lines. Parsing/validating/bogon-filtering that in interpreted + PowerShell is the slow part (minutes), so the hot loop is compiled once via Add-Type (C#) -- it runs in + ~1s. Downloads stream with a live Write-Progress bar; every phase prints its own elapsed time so you can + see exactly where the wall-clock goes. + + Requires PowerShell 7 (pwsh), which runs on Windows, Linux and macOS -- Windows PowerShell 5.1 is + not supported and the script refuses to run there. Schedule it with Task Scheduler, cron, or a + systemd timer. + + Every run rewrites the whole file, so an IP that drops off the feeds stops being blocked on the next + run -- there is no TTL to tune. Calling it is idempotent: if the list on disk is younger than + -MinInterval the script exits without downloading anything, so an over-eager trigger costs nothing + upstream. -Force overrides that. + +.PARAMETER DistributionPath + Path to the shard's Distribution folder. The blocklist is written to the Configuration/ip-blocklist.txt + beneath it, which is the default `file` in blocklist.json. Not needed when the script is run from its + place in the repo (tools/), or when -OutFile is given. + +.PARAMETER OutFile + Explicit output path, overriding -DistributionPath. Use this if you relocated the blocklist and + changed `file` in blocklist.json to match. + +.PARAMETER MinInterval + Refuse to re-run while the existing blocklist is younger than this (default 2h), so a misbehaving + scheduler, a login script or a stuck retry loop cannot hammer the upstream feeds. The age comes from + the `generated=` header of the file already on disk (falling back to its mtime), so it survives across + machines and reboots -- there is no separate state file. Nothing is downloaded when the check trips. + Accepts `90s`, `45m`, `2h`, `2.5h`, `1d`, or a bare number of hours. Use `0` to disable the check. + Match this to how often you actually want fresh data: the anchor feed only refreshes every 2h, so + running more often than that costs bandwidth and gains nothing. + +.PARAMETER Force + Run regardless of how recently the blocklist was generated (bypasses -MinInterval). + +.PARAMETER Feeds + Which feeds to include (by Name). Default: all of them. + +.PARAMETER ExcludeAnonymizers + Also download Tor-exit / open-proxy / VPN-tunnel lists and SUBTRACT those IPs from the output. Off by + default -- for a game server, Tor/open-proxy relays are attack infrastructure you want to block. Turn + this on only if you need to keep VPN/Tor players reachable. + +.PARAMETER DryRun + Download + parse + merge + count only. Writes nothing. + +.EXAMPLE + .\Export-IpBlocklist.ps1 -DryRun + +.EXAMPLE + # Safe to call as often as you like -- it no-ops unless the list is older than 2h. + .\Export-IpBlocklist.ps1 -DistributionPath 'C:\Shard\Distribution' + +.EXAMPLE + .\Export-IpBlocklist.ps1 -OutFile 'D:\shared\ip-blocklist.txt' -ExcludeAnonymizers + +.EXAMPLE + # Regenerate right now, ignoring the cooldown. + .\Export-IpBlocklist.ps1 -DistributionPath 'C:\Shard\Distribution' -Force + +.EXAMPLE + # Linux/macOS, e.g. from cron: + pwsh -File /opt/modernuo/Export-IpBlocklist.ps1 -DistributionPath /opt/modernuo/Distribution + +.NOTES + Feeds are aggressive-but-low-FP for a game server (attacker / botnet / compromised / abuse-relay SOURCE + IPs). Reserved/bogon space (0/8, 10/8, 127/8, RFC1918, multicast, etc.) is always filtered out -- this + matters because firehol-level1 ships bogon netblocks that would otherwise block private/reserved ranges. +#> +[CmdletBinding()] +param( + [string] $DistributionPath, + [string] $OutFile, + [string] $MinInterval = '2h', + [string[]] $Feeds, + [switch] $ExcludeAnonymizers, + [switch] $Force, + [switch] $DryRun +) + +$ErrorActionPreference = 'Stop' +$UA = 'ModernUO-Blocklist-Export' +$totalSw = [System.Diagnostics.Stopwatch]::StartNew() + +# Default location under the Distribution folder. Keep in sync with BlocklistSettings.File. +# Kept as separate segments (never a literal 'a\b') so Join-Path picks the right separator per OS. +$DefaultPathSegments = @('Configuration', 'ip-blocklist.txt') + +# --------------------------------------------------------------------------------------------------------- +# Resolve the output path. Explicit -OutFile wins; then -DistributionPath; then the in-repo layout +# (tools\ sits next to Distribution\) so a checkout works with no arguments at all. The script is meant to +# be copied onto the shard host, and there it needs -DistributionPath (or -OutFile). +# --------------------------------------------------------------------------------------------------------- +if (-not $OutFile) { + if (-not $DistributionPath -and $PSScriptRoot) { + $inRepo = Join-Path (Split-Path -Parent $PSScriptRoot) 'Distribution' + if (Test-Path -LiteralPath $inRepo -PathType Container) { $DistributionPath = $inRepo } + } + if (-not $DistributionPath) { + throw "Could not locate the shard's Distribution folder. Pass -DistributionPath 'C:\path\to\Distribution' (or -OutFile)." + } + if (-not (Test-Path -LiteralPath $DistributionPath -PathType Container)) { + throw "DistributionPath '$DistributionPath' does not exist." + } + $OutFile = Join-Path $DistributionPath @DefaultPathSegments +} + +# --------------------------------------------------------------------------------------------------------- +# Cooldown gate. Runs BEFORE anything is downloaded: the whole point is that a misconfigured scheduler or a +# retry loop cannot spam the upstream feeds. State lives in the output file itself (`generated=` header, +# mtime as fallback), so it is correct across reboots, machines and hand-runs with no sidecar state file. +# --------------------------------------------------------------------------------------------------------- +function ConvertTo-Duration { + param([string]$Text) + if ([string]::IsNullOrWhiteSpace($Text)) { return [TimeSpan]::Zero } + $t = $Text.Trim().ToLowerInvariant() + $unit = $t[$t.Length - 1] + $numText = if ($unit -match '[0-9.]') { $t } else { $t.Substring(0, $t.Length - 1) } + $n = 0.0 + # InvariantCulture is not optional here: under a comma-decimal locale (de-DE, fr-FR, ...) the + # current-culture parse reads '2.5' as 25 -- it treats '.' as a group separator and SUCCEEDS, so + # `-MinInterval 2.5h` would silently become a 25 hour cooldown instead of failing loudly. + if (-not [double]::TryParse($numText, [Globalization.NumberStyles]::Float, + [Globalization.CultureInfo]::InvariantCulture, [ref]$n)) { + throw "Could not parse duration '$Text' (try 90s, 45m, 2h, 2.5h, 1d)." + } + switch ($unit) { + 's' { return [TimeSpan]::FromSeconds($n) } + 'm' { return [TimeSpan]::FromMinutes($n) } + 'h' { return [TimeSpan]::FromHours($n) } + 'd' { return [TimeSpan]::FromDays($n) } + default { return [TimeSpan]::FromHours($n) } # bare number == hours + } +} + +# Age of the list already on disk, or $null when there is nothing usable to age. +function Get-BlocklistAge { + param([string]$Path) + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { return $null } + + # Prefer the header we wrote: it describes the data, not the file, so copying/restoring the file + # cannot make a stale list look fresh (or a fresh one look stale). + try { + $first = Get-Content -LiteralPath $Path -TotalCount 1 -ErrorAction Stop + if ($first -and $first.StartsWith('#')) { + foreach ($tok in $first.Split(' ', [StringSplitOptions]::RemoveEmptyEntries)) { + if ($tok.StartsWith('generated=', [StringComparison]::Ordinal)) { + $stamp = [DateTime]::MinValue + $styles = [Globalization.DateTimeStyles]::AdjustToUniversal -bor [Globalization.DateTimeStyles]::AssumeUniversal + if ([DateTime]::TryParse($tok.Substring(10), [Globalization.CultureInfo]::InvariantCulture, $styles, [ref]$stamp)) { + return @{ Age = ([DateTime]::UtcNow - $stamp); Stamp = $tok.Substring(10); Source = 'header' } + } + } + } + } + } + catch { } + + # Hand-maintained or truncated file: fall back to the filesystem timestamp. + try { + $w = (Get-Item -LiteralPath $Path -ErrorAction Stop).LastWriteTimeUtc + return @{ Age = ([DateTime]::UtcNow - $w) + Stamp = $w.ToString('yyyy-MM-ddTHH:mm:ssZ', [Globalization.CultureInfo]::InvariantCulture) + Source = 'mtime' } + } + catch { return $null } +} + +$minAge = ConvertTo-Duration $MinInterval +if (-not $Force -and $minAge -gt [TimeSpan]::Zero) { + $existing = Get-BlocklistAge -Path $OutFile + if ($existing) { + # A negative age means the stamp is in the future (clock skew, or a file from another host). Treat it + # as fresh: refusing to run is the recoverable failure, hammering the feeds on every tick is not. + if ($existing.Age -lt $minAge) { + $agoText = if ($existing.Age -lt [TimeSpan]::Zero) { 'in the future -- check the clock' } else { ("{0:N1}h ago" -f $existing.Age.TotalHours) } + Write-Host ("Blocklist at {0} was generated {1} ({2}={3}); newer than -MinInterval {4}." -f ` + $OutFile, $agoText, $existing.Source, $existing.Stamp, $MinInterval) + Write-Host "Nothing downloaded. Pass -Force to regenerate now, or lower -MinInterval." + return + } + } +} + +# --------------------------------------------------------------------------------------------------------- +# Compiled hot loop. Interpreted PowerShell chokes on bitwire's ~4M lines; this parses + validates + bogon- +# filters + de-dupes in one compiled pass, and writes the final file directly (no 4M-element PS pipelines). +# Deliberately plain C#: no LINQ, no generics beyond HashSet, nothing that would slow the hot loop. +# --------------------------------------------------------------------------------------------------------- +Add-Type -TypeDefinition @' +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +public static class BlocklistExporter +{ + static bool TryParseIPv4(string s, int start, int len, out uint val) + { + val = 0; + uint acc = 0; int octet = 0, dots = 0, digits = 0, end = start + len; + for (int i = start; i < end; i++) + { + char c = s[i]; + if (c == '.') + { + if (digits == 0 || octet > 255) return false; + acc = (acc << 8) | (uint)octet; dots++; octet = 0; digits = 0; + } + else if (c >= '0' && c <= '9') + { + octet = octet * 10 + (c - '0'); if (++digits > 3) return false; + } + else return false; + } + if (dots != 3 || digits == 0 || octet > 255) return false; + val = (acc << 8) | (uint)octet; + return true; + } + + static bool IsBogon(uint start, uint end, uint[] bs, uint[] be) + { + for (int i = 0; i < bs.Length; i++) + if (start <= be[i] && end >= bs[i]) return true; + return false; + } + + // Parse one feed's text; add bare IPs to `singles`, CIDRs to `cidrs`. Returns count newly added. + public static int AddContent(string content, HashSet singles, HashSet cidrs, uint[] bs, uint[] be) + { + int added = 0, n = content.Length, i = 0; + while (i < n) + { + int eol = content.IndexOf('\n', i); + int lineEnd = (eol < 0) ? n : eol; + int a = i, b = lineEnd; + while (a < b && (content[a] == ' ' || content[a] == '\t' || content[a] == '\r')) a++; + while (b > a && (content[b - 1] == ' ' || content[b - 1] == '\t' || content[b - 1] == '\r')) b--; + i = (eol < 0) ? n : eol + 1; + if (a >= b) continue; + char first = content[a]; + if (first == '#' || first == ';') continue; + + // Feeds vary: some are bare IPs, some are CSV/whitespace records with the IP first. + int t = a; + while (t < b) + { + char c = content[t]; + if (c == ' ' || c == '\t' || c == ',' || c == ';') break; + t++; + } + int slash = -1; + for (int k = a; k < t; k++) { if (content[k] == '/') { slash = k; break; } } + + if (slash >= 0) + { + uint ip; + if (!TryParseIPv4(content, a, slash - a, out ip)) continue; + int bits = 0, bd = 0; + for (int k = slash + 1; k < t; k++) + { + char c = content[k]; + if (c < '0' || c > '9') { bd = -1; break; } + bits = bits * 10 + (c - '0'); bd++; + } + if (bd <= 0 || bits > 32) continue; + ulong size = (bits == 0) ? 0xFFFFFFFFUL : ((1UL << (32 - bits)) - 1UL); + ulong endAddr = (ulong)ip + size; if (endAddr > 0xFFFFFFFFUL) endAddr = 0xFFFFFFFFUL; + if (IsBogon(ip, (uint)endAddr, bs, be)) continue; + if (cidrs.Add(content.Substring(a, t - a))) added++; + } + else + { + uint ip; + if (!TryParseIPv4(content, a, t - a, out ip)) continue; + if (IsBogon(ip, ip, bs, be)) continue; + if (singles.Add(ip)) added++; + } + } + return added; + } + + static int WriteOctet(char[] buf, int pos, uint v) + { + if (v >= 100) { buf[pos++] = (char)('0' + v / 100); buf[pos++] = (char)('0' + (v / 10) % 10); } + else if (v >= 10) { buf[pos++] = (char)('0' + v / 10); } + buf[pos++] = (char)('0' + v % 10); + return pos; + } + + // Writes the whole blocklist in one streamed pass so we never materialize millions of strings. + // Header first (the shard's reload detector requires it), then singles, then CIDRs. LF line endings. + public static void Write(string path, string header, HashSet singles, HashSet cidrs) + { + using (var w = new StreamWriter(path, false, new UTF8Encoding(false), 1 << 20)) + { + w.Write(header); w.Write('\n'); + + char[] buf = new char[20]; + foreach (uint v in singles) + { + int p = 0; + p = WriteOctet(buf, p, (v >> 24) & 255); buf[p++] = '.'; + p = WriteOctet(buf, p, (v >> 16) & 255); buf[p++] = '.'; + p = WriteOctet(buf, p, (v >> 8) & 255); buf[p++] = '.'; + p = WriteOctet(buf, p, v & 255); buf[p++] = '\n'; + w.Write(buf, 0, p); + } + + foreach (string c in cidrs) { w.Write(c); w.Write('\n'); } + } + } +} +'@ + +# --------------------------------------------------------------------------------------------------------- +# Feed set -- thin, non-overlapping. See .DESCRIPTION for why each is kept and what was dropped as redundant. +# romainmarcoux's "full" set is sharded; only aa..ad carry data today (ae.. are empty placeholders). +# A 404/empty shard is skipped, so extend this list if upstream grows the shard count. +# --------------------------------------------------------------------------------------------------------- +$rmBase = 'https://raw.githubusercontent.com/romainmarcoux/malicious-ip/main/full-300k-' +$rmShards = @('aa','ab','ac','ad') | ForEach-Object { $rmBase + $_ + '.txt' } + +$AllFeeds = @( + [pscustomobject]@{ Name = 'bitwire-it'; Urls = @('https://raw.githubusercontent.com/bitwire-it/ipblocklist/main/inbound.txt') } + [pscustomobject]@{ Name = 'romainmarcoux'; Urls = $rmShards } + [pscustomobject]@{ Name = 'sentinel-turris';Urls = @('https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv') } + [pscustomobject]@{ Name = 'firehol-level1'; Urls = @('https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset') } +) + +# Anonymizer / relay lists subtracted only when -ExcludeAnonymizers is set (Tor exits, open proxies, VPN tunnels). +$AnonFeeds = @( + 'https://raw.githubusercontent.com/borestad/firehol-mirror/refs/heads/main/tor_exits.ipset' + 'https://raw.githubusercontent.com/borestad/firehol-mirror/refs/heads/main/sslproxies_7d.ipset' + 'https://raw.githubusercontent.com/borestad/firehol-mirror/refs/heads/main/socks_proxy_7d.ipset' + 'https://raw.githubusercontent.com/ShadowWhisperer/IPs/master/Lists/Tunnels' +) + +if ($Feeds) { + $AllFeeds = $AllFeeds | Where-Object { $Feeds -contains $_.Name } + if (-not $AllFeeds) { throw "No feeds matched -Feeds." } +} + +# --------------------------------------------------------------------------------------------------------- +# Reserved / bogon ranges -- never valid attacker SOURCE IPs; always filtered. Built once as uint32 arrays. +# --------------------------------------------------------------------------------------------------------- +function ConvertTo-IPv4UInt { + param([string]$s) + $a = $s.Split('.') + if ($a.Length -ne 4) { return $null } + $v = [uint32]0 + foreach ($o in $a) { + $n = 0 + if (-not [int]::TryParse($o, [ref]$n) -or $n -lt 0 -or $n -gt 255) { return $null } + $v = ($v -shl 8) -bor [uint32]$n + } + return $v +} + +$bogonCidrs = '0.0.0.0/8','10.0.0.0/8','100.64.0.0/10','127.0.0.0/8','169.254.0.0/16','172.16.0.0/12', + '192.0.0.0/24','192.0.2.0/24','192.168.0.0/16','198.18.0.0/15','198.51.100.0/24', + '203.0.113.0/24','224.0.0.0/3' # 224/3 covers multicast + reserved + 255.255.255.255 +$bogStart = [System.Collections.Generic.List[uint32]]::new() +$bogEnd = [System.Collections.Generic.List[uint32]]::new() +foreach ($c in $bogonCidrs) { + $p = $c.Split('/'); $base = ConvertTo-IPv4UInt $p[0]; $bits = [int]$p[1] + $size = [uint32]([Math]::Pow(2, 32 - $bits)) + $bogStart.Add($base); $bogEnd.Add([uint32]($base + $size - 1)) +} +$bogStart = $bogStart.ToArray(); $bogEnd = $bogEnd.ToArray() + +# --------------------------------------------------------------------------------------------------------- +# Streaming download with a live progress bar (Write-Progress) so large feeds show real byte progress. +# --------------------------------------------------------------------------------------------------------- +function Get-Url { + param([string]$Url, [string]$Label) + $req = [System.Net.HttpWebRequest]::Create($Url) + $req.UserAgent = $UA; $req.Timeout = 120000; $req.ReadWriteTimeout = 120000 + $resp = $req.GetResponse() + try { + $total = $resp.ContentLength + $stream = $resp.GetResponseStream() + $ms = New-Object System.IO.MemoryStream + $buf = New-Object byte[] (1MB) + $read = 0; $lastReport = 0 + while (($n = $stream.Read($buf, 0, $buf.Length)) -gt 0) { + $ms.Write($buf, 0, $n); $read += $n + if ($read - $lastReport -ge 2MB) { + $lastReport = $read + if ($total -gt 0) { + Write-Progress -Activity ("Downloading {0}" -f $Label) -PercentComplete ([int](100 * $read / $total)) ` + -Status ("{0:N1} / {1:N1} MB" -f ($read / 1MB), ($total / 1MB)) + } else { + Write-Progress -Activity ("Downloading {0}" -f $Label) -Status ("{0:N1} MB" -f ($read / 1MB)) + } + } + } + Write-Progress -Activity ("Downloading {0}" -f $Label) -Completed + return [System.Text.Encoding]::UTF8.GetString($ms.ToArray()) + } + finally { $resp.Close() } +} + +# --------------------------------------------------------------------------------------------------------- +# Collect every kept feed into ONE global set, timing each phase. +# --------------------------------------------------------------------------------------------------------- +$singles = [System.Collections.Generic.HashSet[uint32]]::new() +$cidrs = [System.Collections.Generic.HashSet[string]]::new() +$feedCount = 0 + +foreach ($feed in $AllFeeds) { + $before = $singles.Count + $cidrs.Count + $ok = $false + foreach ($url in $feed.Urls) { + try { + $dlSw = [System.Diagnostics.Stopwatch]::StartNew() + $content = Get-Url -Url $url -Label $feed.Name + $dlSw.Stop() + $mb = [Math]::Round($content.Length / 1MB, 1) + + $pSw = [System.Diagnostics.Stopwatch]::StartNew() + [void][BlocklistExporter]::AddContent($content, $singles, $cidrs, $bogStart, $bogEnd) + $pSw.Stop() + Write-Host (" [dl {0,6:N1}s / parse {1,5:N1}s] {2}" -f $dlSw.Elapsed.TotalSeconds, $pSw.Elapsed.TotalSeconds, ("{0} ({1} MB)" -f $feed.Name, $mb)) + $ok = $true + } + catch { Write-Warning ("{0}: {1} -- skipping shard ({2})" -f $feed.Name, $url, $_.Exception.Message) } + } + if ($ok) { + $feedCount++ + $delta = ($singles.Count + $cidrs.Count) - $before + Write-Host ("{0,-16} +{1,8} new (running total {2} ip / {3} cidr)`n" -f $feed.Name, $delta, $singles.Count, $cidrs.Count) + } + else { Write-Warning ("{0}: all sources failed -- skipping" -f $feed.Name) } +} + +# --------------------------------------------------------------------------------------------------------- +# Optional: subtract Tor / open-proxy / VPN IPs. +# --------------------------------------------------------------------------------------------------------- +if ($ExcludeAnonymizers) { + $anon = [System.Collections.Generic.HashSet[uint32]]::new() + $anonCidr = [System.Collections.Generic.HashSet[string]]::new() + foreach ($url in $AnonFeeds) { + try { [void][BlocklistExporter]::AddContent((Get-Url -Url $url -Label 'anonymizers'), $anon, $anonCidr, $bogStart, $bogEnd) } + catch { Write-Warning ("anonymizer list {0}: {1}" -f $url, $_.Exception.Message) } + } + $removed = 0 + foreach ($ip in @($anon)) { if ($singles.Remove($ip)) { $removed++ } } + Write-Host ("ExcludeAnonymizers: removed {0} Tor/proxy/VPN single IPs" -f $removed) +} + +$total = $singles.Count + $cidrs.Count +Write-Host ("Merged {0} feed(s): {1} unique single IPs + {2} unique CIDRs (bogon-filtered) in {3:N1}s." -f ` + $feedCount, $singles.Count, $cidrs.Count, $totalSw.Elapsed.TotalSeconds) + +if ($DryRun) { + Write-Host ("DRY RUN: nothing written (would have written {0} entries to {1})." -f $total, $OutFile) + return +} + +# A partial feed outage must not silently shrink the shard's blocklist to nothing; keep the last good file. +if ($total -eq 0) { throw "No entries parsed -- refusing to overwrite '$OutFile' with an empty list." } + +# --------------------------------------------------------------------------------------------------------- +# Write to a .tmp sibling and swap it into place, so the shard (which reads the whole file on a change) +# never observes a half-written list. One rename does it whether or not a list is already there. +# --------------------------------------------------------------------------------------------------------- +$outDir = Split-Path -Parent $OutFile +if ($outDir -and -not (Test-Path -LiteralPath $outDir -PathType Container)) { + New-Item -ItemType Directory -Path $outDir -Force | Out-Null +} + +# InvariantCulture: ':' is the culture-defined time separator in a custom format string, and the +# header is a machine-read marker the shard compares verbatim. +$generated = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ', [Globalization.CultureInfo]::InvariantCulture) +$header = "# modernuo-blocklist generated=$generated count=$total ipv4=$($singles.Count) cidr=$($cidrs.Count) feeds=$feedCount" + +$tmp = $OutFile + '.tmp' +$wSw = [System.Diagnostics.Stopwatch]::StartNew() +try { + [BlocklistExporter]::Write($tmp, $header, $singles, $cidrs) + # One atomic rename over the destination on every platform: MoveFileEx REPLACE_EXISTING on + # Windows, rename(2) on Linux and macOS. + [IO.File]::Move($tmp, $OutFile, $true) +} +finally { + # Never leave a partial .tmp next to a live blocklist for the next run to trip over. + if (Test-Path -LiteralPath $tmp -PathType Leaf) { Remove-Item -LiteralPath $tmp -Force -ErrorAction SilentlyContinue } +} +$wSw.Stop() + +$sizeMb = [Math]::Round((Get-Item -LiteralPath $OutFile).Length / 1MB, 1) +Write-Host ("`nWrote {0} entries ({1} MB) to {2} in {3:N1}s (total {4:N1}s). generated={5}" -f ` + $total, $sizeMb, $OutFile, $wSw.Elapsed.TotalSeconds, $totalSw.Elapsed.TotalSeconds, $generated) +Write-Host "The shard picks this up on its next reloadInterval poll; no restart needed."