Two features ran on every shard out of the box, each polling on its own 60s timer for files most shards never generate, neither ever asked for. Fixing that turned into untangling why they shared a config file — and then into the on-loop cost of the three lists behind them. ## Before / after Measured on the shipped defaults. On-loop numbers are what freezes the world; the tick budget is 8 ms. | | before | after | |---|---:|---:| | Blocklist poll on a shard with no list | every 60s, forever | **none** (opt-in) | | Manual allowlist poll on a shard with no carve-outs | every 60s, forever | **none** (opt-in) | | Promote-guard sweep timer | leaked on `Stop()` | stopped, and only started when hits are reported | | Login allowlist flush, on-loop | O(n) walk + 2 arrays **every 60s**, LOH past ~5,300 entries | reused buffers, **hourly**, zero steady-state allocation | | Auto-denylist, accept path | 9.1 ns/call | **6.1 ns/call** | | Auto-denylist, sustained flood at cap (60k rejected) | 26.7 ms | **9.3 ms** | | Auto-denylist, flood end — **worst single call** | 9.49 ms | **0.05 ms** | | Auto-denylist cap | 65,536 (stranding 9,895 slots) | **324,449** (exact `HashSet` capacity, ~19 MB) | The auto-denylist row that matters is the third: the on-loop stall at flood end drops **190×**, because retiring lapsed holds is now the number expiring rather than the number held. ## Why this design It is built for the shape of attack these shards actually see: **hundreds to a few thousand connections per second**, occasionally tens of thousands, sustained over minutes rather than delivered instantly. Against that shape the cap now covers the whole observed range (50k–250k distinct sources) in memory, and the work of expiring them spreads across the accept calls that were already happening. There is one case this design is *worse* at than the old one: if every held entry lapses within the same millisecond, retiring them costs ~10.7 ms against the old ~8.9 ms, because the ring's random-access set removals lose to a sequential dictionary scan. Reaching it requires an entire flood to arrive inside one millisecond. **A shard absorbing 324,449 connections in a millisecond is finished at the accept path no matter what this list does** — that is the point where the answer is upstream security and scrubbing (an L4 proxy, edge filtering, a bouncer at the kernel), not a data structure in the game loop. We chose the design that fits the attacks we see and degrades honestly past them, rather than over-engineering for one we do not. ## Blocklist — now opt-in `BlocklistFilter.Start` only bailed when `_path == null`, which needs `file` to be empty. The default is `"Configuration/ip-blocklist.txt"`, so on any default install both `Task.Run(PollLoop)` and a recurring `SweepGuard` timer started unconditionally, logging *"Blocklist inert: no list at …; polling every 60s"* and then doing exactly that forever. Adds `"enabled"`, default `false`, using the `_enabled = s.Enabled && <preconditions>` idiom already in `LoginAllowlist` and `AutoDenylist`. **Upgrade is deliberately loud**: a missing key binds to the default, so `LogWhyDisabled()` splits three cases and a shard with a list on disk but no `enabled` key gets a **Warning**, not silence. ## `FileAllowlist` → `ManualAllowlist`, with its own config Moves to `Configuration/ip-allowlist.json` (`enabled` default `false`, `files`, `reloadInterval`) and into `Network/ManualAllowlist/`, mirroring `Network/LoginAllowlist/`. It was never a sub-feature of the blocklist. `ManualAllowlist.Contains` has two callers: | Caller | Could anything else do it? | |---|---| | `BlocklistFilter.Evaluate` | **Yes** — the generator already subtracts these files at generation time | | `BanExemptions.IsExempt` | **No** — sole mechanism for suppressing behavioural ban contributions | The second reaches `BanChannel.IsExempt` with no blocklist in the path. A shard running **no blocklist** still needs this so the admin's own IP isn't auto-banned by rate-limit detection, so a shared flag couldn't express it — the implication is asymmetric. They still work together via a startup warning when the blocklist is on and the allowlist is not. On the name: "File" described the storage. The distinction from `LoginAllowlist` is **provenance** — declared by an operator versus earned by authenticating — and "Manual" matches `BanReasons.Manual`. `allowlistFiles` is removed from `BlocklistSettings` outright; blocklists have not shipped long enough for anyone to have set it. ## Login allowlist flush `Flush()` allocated two arrays sized to the live entry count and copied the whole dictionary into them **on the game loop**, every 60s. `UInt128` is 16 bytes, so past ~5,300 entries that first array was an LOH allocation once a minute, forever. The file write was already off-loop; the walk was not. Static buffers grown geometrically; the writer owns them until it posts completion back through `Core.LoopContext`, so `_writing`/`_dirty` stay loop state (rule #10). Interval → 1 hour against a 90-day TTL. Clean shutdown writes synchronously via `EventSink.Shutdown`; `HandleClosed` skips `InvokeShutdown` when crashed, so the crash path subscribes separately and only writes when it is actually on the loop thread. Also fixes a pre-existing hole where `_dirty` was cleared *before* the write, so a failed write dropped entries despite the comment promising a retry. ## Auto-denylist: expiry ring Reclaiming lapsed holds was O(entries held) — every cap-triggered reclaim during a flood walked the whole dictionary to find the few that expired, and `_warnedFull` suppressed the log, not the work. A hold is **never refreshed** now: the first detection sets the expiry, later ones leave it. That makes insertion order equal to expiry order, so a ring of the same keys is sorted by construction and retiring stops at the first live record. Nothing is lost — the rate limiter runs *ahead* of the connection filters (`NetState.Network.cs`) and reports to the ban channel, so a flooder whose hold lapses is re-held on its next attempt. Because the ring carries the expiry, the membership side only answers "present?", so it is a `HashSet` — measured at **36 B/slot against the dictionary's 52**. `HashSet` and `Dictionary` share `HashHelpers`, so the from-empty capacity progression is identical (36,353 → 75,431 → 156,437 → 324,449 → 672,827) and the cap still lands on one exactly. The ring is parallel `UInt128[]`/`long[]` rather than an array of structs — `UInt128` forces 16-byte alignment, so a packed pair costs 32 bytes where these cost 24, and the drain reads only the `long[]`. Rejected after measuring: splitting the drain into a scan loop plus a removal loop (inside noise — both issue N hash removes, and the pointer math was never the bottleneck), and `Dictionary<UInt128,bool>` with tombstoning instead of removal (10% slower *and* unbounded, which breaks the cap). ## Testing Build clean, 0 warnings. **1,530 tests pass** — 708 UOContent, 822 Server. Tests were reworked rather than patched: the refresh test inverts to `Repeat_detection_does_not_extend_the_hold`, the obsolete sweep-throttle test is deleted along with the throttle, and four were added for the ring — set/ring parity, release-then-re-hold not being retired by the stale record, exact fill of a non-power-of-two cap, and the moved allowlist config's casing contract. The throttle test added mid-PR was verified to fail without its fix before being deleted. One commit is comments only (verified: a diff filtered of `//` lines is empty), removing development narration — a `"(Task 2)"` plan reference, `"matching the per-feature JSON config pattern used by X"` across four loaders, a duplicated threading note — and repointing `Firewall` at `dev-docs/ip-bans-and-allowlists.md` instead of a "ban-channel design doc" that does not exist. Note `Distribution/Configuration/blocklist.json` is gitignored (`.gitignore:14`) and generated from the record defaults on first boot, so the record default *is* the shipped default.
413 lines
13 KiB
C#
413 lines
13 KiB
C#
/*************************************************************************
|
|
* ModernUO *
|
|
* Copyright 2019-2026 - ModernUO Development Team *
|
|
* Email: hi@modernuo.com *
|
|
* File: Firewall.cs *
|
|
* *
|
|
* This program is free software: you can redistribute it and/or modify *
|
|
* it under the terms of the GNU General Public License as published by *
|
|
* the Free Software Foundation, either version 3 of the License, or *
|
|
* (at your option) any later version. *
|
|
* *
|
|
* You should have received a copy of the GNU General Public License *
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
|
|
*************************************************************************/
|
|
|
|
using System;
|
|
using System.Buffers;
|
|
using System.Collections.Generic;
|
|
using System.IO;
|
|
using System.Net;
|
|
using System.Runtime.CompilerServices;
|
|
using Server.Collections;
|
|
using Server.Json;
|
|
using Server.Logging;
|
|
|
|
namespace Server.Network;
|
|
|
|
public static class Firewall
|
|
{
|
|
// Single-threaded: the accept path, admin gump/command, TTL expiry timer, and boot load all run on
|
|
// the main game loop, so no locks, caches, or version counters are needed. _entries is the
|
|
// authoritative store; _index is a derived, rebuild-on-demand SortedRangeIndex used only for the
|
|
// accept-path IsBlocked lookup, over the same primitive the blocklist uses (see BlocklistSnapshot).
|
|
// See dev-docs/ip-bans-and-allowlists.md.
|
|
private static readonly List<IFirewallEntry> _entries = [];
|
|
|
|
// Entries with a TTL: entry -> absolute expiry tick (Core.TickCount). Permanent entries are absent.
|
|
private static readonly Dictionary<IFirewallEntry, long> _expiring = [];
|
|
|
|
private static SortedRangeIndex<UInt128> _index = SortedRangeIndex<UInt128>.Empty;
|
|
private static bool _indexDirty;
|
|
|
|
private static readonly ILogger logger = LogFactory.GetLogger(typeof(Firewall));
|
|
private const string _path = "Configuration/firewall.json";
|
|
private const string _legacyPath = "firewall.cfg";
|
|
private static bool _dirty;
|
|
private static bool _configured;
|
|
|
|
public static int FirewallSetCount => _entries.Count;
|
|
|
|
public static void ReadFirewallSet(Action<IReadOnlyCollection<IFirewallEntry>> callback) => callback(_entries);
|
|
|
|
public static bool IsBlocked(IPAddress address)
|
|
{
|
|
if (_entries.Count == 0)
|
|
{
|
|
return false;
|
|
}
|
|
|
|
EnsureIndex();
|
|
return _index.Contains(address.ToUInt128());
|
|
}
|
|
|
|
// Rebuilds the derived lookup index from the authoritative _entries list, but only when entries have
|
|
// changed since the last build. Runs on the main game loop, so the pooled build buffer is single-threaded
|
|
// (mt: false); only the two final SortedRangeIndex arrays are heap-allocated.
|
|
private static void EnsureIndex()
|
|
{
|
|
if (!_indexDirty)
|
|
{
|
|
return;
|
|
}
|
|
|
|
using var ranges = PooledRefList<SortedRangeIndex<UInt128>.Range>.Create(_entries.Count, mt: false);
|
|
for (var i = 0; i < _entries.Count; i++)
|
|
{
|
|
var entry = _entries[i];
|
|
ranges.Add(new SortedRangeIndex<UInt128>.Range(entry.MinIpAddress, entry.MaxIpAddress));
|
|
}
|
|
|
|
ranges.Sort(SortedRangeIndex<UInt128>.ByMin);
|
|
_index = SortedRangeIndex<UInt128>.Build(ranges.AsSpan());
|
|
_indexDirty = false;
|
|
}
|
|
|
|
public static bool Add(IFirewallEntry firewallEntry) => Add(firewallEntry, TimeSpan.Zero);
|
|
|
|
/// <summary>
|
|
/// Adds an entry. <paramref name="ttl"/> <= <see cref="TimeSpan.Zero"/> means permanent. Returns false
|
|
/// if the entry was already present.
|
|
/// </summary>
|
|
// Indexed scan (no closure allocation); firewall lists are small, so O(n) is negligible and this
|
|
// stays off the hot path (Add/Remove are admin/boot actions, not the accept path).
|
|
private static int IndexOfEntry(IFirewallEntry entry)
|
|
{
|
|
for (var i = 0; i < _entries.Count; i++)
|
|
{
|
|
if (_entries[i].CompareTo(entry) == 0)
|
|
{
|
|
return i;
|
|
}
|
|
}
|
|
|
|
return -1;
|
|
}
|
|
|
|
public static bool Add(IFirewallEntry firewallEntry, TimeSpan ttl, bool persist = true)
|
|
{
|
|
if (firewallEntry == null || IndexOfEntry(firewallEntry) >= 0)
|
|
{
|
|
return false;
|
|
}
|
|
|
|
_entries.Add(firewallEntry);
|
|
|
|
if (ttl > TimeSpan.Zero)
|
|
{
|
|
_expiring[firewallEntry] = Core.TickCount + (long)ttl.TotalMilliseconds;
|
|
}
|
|
|
|
_indexDirty = true;
|
|
|
|
if (persist)
|
|
{
|
|
MarkDirty();
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
public static bool Remove(IFirewallEntry entry)
|
|
{
|
|
if (entry == null)
|
|
{
|
|
return false;
|
|
}
|
|
|
|
var index = IndexOfEntry(entry);
|
|
if (index < 0)
|
|
{
|
|
return false;
|
|
}
|
|
|
|
// Remove the stored instance from _expiring (not the passed reference), so a value-equal
|
|
// entry created elsewhere still clears the TTL bookkeeping.
|
|
var stored = _entries[index];
|
|
_entries.RemoveAt(index);
|
|
_expiring.Remove(stored);
|
|
_indexDirty = true;
|
|
MarkDirty();
|
|
return true;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Removes every entry whose TTL has elapsed. Called from the main-thread maintenance timer.
|
|
/// </summary>
|
|
internal static void ExpireEntries(long nowTicks)
|
|
{
|
|
if (_expiring.Count == 0)
|
|
{
|
|
return;
|
|
}
|
|
|
|
List<IFirewallEntry> expired = null;
|
|
foreach (var (entry, expiresAt) in _expiring)
|
|
{
|
|
if (expiresAt - nowTicks <= 0)
|
|
{
|
|
(expired ??= []).Add(entry);
|
|
}
|
|
}
|
|
|
|
if (expired == null)
|
|
{
|
|
return;
|
|
}
|
|
|
|
for (var i = 0; i < expired.Count; i++)
|
|
{
|
|
var entry = expired[i];
|
|
_entries.Remove(entry);
|
|
_expiring.Remove(entry);
|
|
}
|
|
|
|
_indexDirty = true;
|
|
MarkDirty();
|
|
}
|
|
|
|
[MethodImpl(MethodImplOptions.AggressiveInlining)]
|
|
public static IFirewallEntry ToFirewallEntry(object entry) =>
|
|
entry switch
|
|
{
|
|
IFirewallEntry firewallEntry => firewallEntry,
|
|
IPAddress address => new SingleIpFirewallEntry(address),
|
|
string s => ToFirewallEntry(s),
|
|
_ => null
|
|
};
|
|
|
|
public static IFirewallEntry ToFirewallEntry(string entry)
|
|
{
|
|
if (entry == null)
|
|
{
|
|
return null;
|
|
}
|
|
|
|
try
|
|
{
|
|
var rangeSeparator = entry.IndexOf('-');
|
|
if (rangeSeparator > -1)
|
|
{
|
|
return new CidrFirewallEntry(
|
|
IPAddress.Parse(entry.AsSpan(0, rangeSeparator)),
|
|
IPAddress.Parse(entry.AsSpan(rangeSeparator + 1))
|
|
);
|
|
}
|
|
|
|
if (entry.IndexOf('/') > -1)
|
|
{
|
|
return new CidrFirewallEntry(entry);
|
|
}
|
|
|
|
return new SingleIpFirewallEntry(entry);
|
|
}
|
|
catch
|
|
{
|
|
return null;
|
|
}
|
|
}
|
|
|
|
public static void Configure()
|
|
{
|
|
if (_configured)
|
|
{
|
|
return;
|
|
}
|
|
_configured = true;
|
|
|
|
var path = Path.Join(Core.BaseDirectory, _path);
|
|
|
|
if (File.Exists(path))
|
|
{
|
|
LoadFrom(JsonConfig.Deserialize<FirewallSettings>(path));
|
|
}
|
|
else
|
|
{
|
|
var legacyPath = ResolveLegacyCfgPath();
|
|
if (legacyPath != null)
|
|
{
|
|
MigrateLegacyCfg(legacyPath);
|
|
Save(); // materialize firewall.json; the .cfg is no longer read after this
|
|
TryMarkLegacyCfgMigrated(legacyPath);
|
|
}
|
|
}
|
|
|
|
// Main-thread maintenance: expire TTLs and flush pending writes. No background thread.
|
|
Timer.DelayCall(TimeSpan.FromSeconds(30), TimeSpan.FromSeconds(30), Maintenance);
|
|
|
|
// Expose the set to the accept path. Everything else (gump, commands, persistence) keeps using
|
|
// the Firewall API directly; only the per-connection question goes through the filter registry.
|
|
ConnectionFilters.Register(FirewallConnectionFilter.Instance);
|
|
}
|
|
|
|
private static void Maintenance()
|
|
{
|
|
ExpireEntries(Core.TickCount);
|
|
|
|
if (_dirty)
|
|
{
|
|
Save();
|
|
}
|
|
}
|
|
|
|
private static void MarkDirty() => _dirty = true;
|
|
|
|
internal static void LoadFrom(FirewallSettings settings)
|
|
{
|
|
if (settings?.Entries == null)
|
|
{
|
|
return;
|
|
}
|
|
|
|
// Core.Now: this runs on the game loop, via the Configure sweep.
|
|
var now = Core.Now;
|
|
var records = settings.Entries;
|
|
for (var i = 0; i < records.Length; i++)
|
|
{
|
|
var record = records[i];
|
|
var entry = ToFirewallEntry(record.Value);
|
|
if (entry == null)
|
|
{
|
|
logger.Warning("Ignoring unparseable firewall entry \"{Entry}\"", record.Value);
|
|
continue;
|
|
}
|
|
|
|
var ttl = TimeSpan.Zero;
|
|
if (record.Expires is { } expires)
|
|
{
|
|
ttl = expires - now;
|
|
if (ttl <= TimeSpan.Zero)
|
|
{
|
|
continue; // already expired
|
|
}
|
|
}
|
|
|
|
Add(entry, ttl, persist: false);
|
|
}
|
|
}
|
|
|
|
internal static FirewallSettings ToSettings()
|
|
{
|
|
// expires is derived below as now + (expiresAtTick - nowTicks), so both operands must come from
|
|
// the same instant. Core.Now and Core.TickCount are refreshed together each loop iteration; a
|
|
// fresh DateTime.UtcNow here would bake the loop's lag into every persisted expiry.
|
|
var now = Core.Now;
|
|
var nowTicks = Core.TickCount;
|
|
var list = new List<FirewallEntryRecord>(_entries.Count);
|
|
|
|
for (var i = 0; i < _entries.Count; i++)
|
|
{
|
|
var entry = _entries[i];
|
|
DateTime? expires = null;
|
|
if (_expiring.TryGetValue(entry, out var expiresAtTick))
|
|
{
|
|
expires = now.AddMilliseconds(expiresAtTick - nowTicks);
|
|
}
|
|
|
|
list.Add(new FirewallEntryRecord { Value = entry.ToString(), Expires = expires });
|
|
}
|
|
|
|
return new FirewallSettings { Entries = list.ToArray() };
|
|
}
|
|
|
|
public static void Save()
|
|
{
|
|
_dirty = false;
|
|
var path = Path.Join(Core.BaseDirectory, _path);
|
|
var tmp = $"{path}.tmp";
|
|
JsonConfig.Serialize(tmp, ToSettings());
|
|
File.Move(tmp, path, overwrite: true); // atomic swap
|
|
}
|
|
|
|
/// <summary>
|
|
/// Locates the legacy firewall.cfg to migrate. The modern convention is <see cref="Core.BaseDirectory"/>,
|
|
/// checked first; the pre-collapse <c>AdminFirewall</c> used a bare relative path (resolved against the
|
|
/// process's current working directory), which may differ from <see cref="Core.BaseDirectory"/> when the
|
|
/// shard is launched from elsewhere, so that's checked as a fallback. Returns null if neither exists.
|
|
/// </summary>
|
|
private static string ResolveLegacyCfgPath()
|
|
{
|
|
var underBaseDirectory = Path.Join(Core.BaseDirectory, _legacyPath);
|
|
if (File.Exists(underBaseDirectory))
|
|
{
|
|
return underBaseDirectory;
|
|
}
|
|
|
|
return File.Exists(_legacyPath) ? _legacyPath : null;
|
|
}
|
|
|
|
private static void MigrateLegacyCfg(string legacyPath)
|
|
{
|
|
var searchValues = SearchValues.Create("*Xx?");
|
|
|
|
using var reader = new StreamReader(legacyPath);
|
|
while (reader.ReadLine() is { } line)
|
|
{
|
|
line = line.Trim();
|
|
if (line.Length == 0)
|
|
{
|
|
continue;
|
|
}
|
|
|
|
if (line.AsSpan().ContainsAny(searchValues))
|
|
{
|
|
logger.Warning("Legacy firewall entry \"{Entry}\" ignored during migration", line);
|
|
continue;
|
|
}
|
|
|
|
var entry = ToFirewallEntry(line);
|
|
if (entry != null)
|
|
{
|
|
Add(entry, TimeSpan.Zero, persist: false);
|
|
}
|
|
}
|
|
|
|
logger.Information("Migrated {Count} entr(ies) from legacy firewall.cfg to firewall.json", _entries.Count);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Renames the migrated <c>.cfg</c> to <c>firewall.cfg.migrated</c> so it isn't re-scanned on the next
|
|
/// boot and operators can see it was already migrated. Best-effort: a locked/read-only file must not
|
|
/// fail startup, since the migration itself (firewall.json) already succeeded.
|
|
/// </summary>
|
|
private static void TryMarkLegacyCfgMigrated(string legacyPath)
|
|
{
|
|
try
|
|
{
|
|
File.Move(legacyPath, $"{legacyPath}.migrated", overwrite: true);
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
logger.Warning(e, "Could not rename migrated legacy firewall file \"{Path}\"", legacyPath);
|
|
}
|
|
}
|
|
|
|
internal static void ResetForTesting()
|
|
{
|
|
_entries.Clear();
|
|
_expiring.Clear();
|
|
_index = SortedRangeIndex<UInt128>.Empty;
|
|
_indexDirty = false;
|
|
_configured = false;
|
|
}
|
|
}
|