ModernUO/Projects/Server/Main.cs
Kamron Batman c39454137e
feat(network): pluggable connection filters; file blocklist + contribute-first CrowdSec (#2542)
Reshapes IP banning around one idea: **core owns the question, content owns every answer.**

Core gains a single accept-path seam — `IConnectionFilter` — and loses everything that used to implement one. The firewall moves to UOContent, a new file-backed blocklist joins it there, and CrowdSec is repositioned from an in-app enforcer to a contribute-first reporter.

## The seam

```csharp
public interface IConnectionFilter
{
    string Name { get; }
    void Configure();
    void Start(CancellationToken token);
    void Stop();
    bool ShouldDeny(IPAddress address);
}
```

The accept path went from hardcoded branches to one question:

```csharp
else if (ConnectionFilters.ShouldDeny(remoteIP, out var deniedBy))
{
    logger.Debug("{Address} denied by connection filter '{Filter}'", remoteIP, deniedBy);
}
```

Filters register during the Configure sweep. The registry is a plain array walked by an indexed loop — no enumerator, no closure, no allocation — and the first denial short-circuits. An interface dispatch is noise next to the `accept()` syscall, so pluggability costs nothing measurable on the path that has to survive a DDoS.

Whatever a hit implies — persisting, promoting to an OS bouncer, contributing to the ban channel — is the filter's business, not the accept path's.

A filter that throws is **unregistered and the connection fails open**. A filter that faults once faults for every subsequent connection, so leaving it registered means an exception and a log line per accept — exactly the amplification an attacker wants — and a broken filter must not be able to deny everyone either.

This deliberately does **not** reuse `EventSink.InvokeSocketConnect`: that fires later and allocates a `SocketConnectEventArgs` per connection, which is what the accept path avoids for rejected traffic.

## What ships behind it

**`firewall`** (UOContent) — the existing admin-curated set. Collapsed from `Firewall` + `AdminFirewall` + a threaded enforcer into one single-threaded store with **zero concurrency primitives**: the accept path, admin gump, TTL expiry and boot load all run on the game loop. Persists to `Configuration/firewall.json` with automatic migration from the legacy `firewall.cfg`. No behavior change for operators — same namespace, same gump, same commands.

**`blocklist`** (UOContent) — new. Holds a millions-strong list in-app and **demand-pages** hits up to CrowdSec, which promotes them to the OS firewall.

The motivation is concrete: CrowdSec's Windows bouncer cannot load the ~3.9M IPs that 91 community feeds produce, but it handles ~100k fine. So the millions live in-process behind a binary search, and only addresses that *actually connect* get promoted. A `PromotedGuard` suppresses re-reporting an address until the bouncer picks it up.

The list is parsed straight from UTF-8 file bytes with no per-line string allocation, off the game loop, and published as an immutable snapshot swapped through a single `volatile` reference. Reloads yield to world saves.

**`tools/Export-IpBlocklist.ps1`** — the producer. Requires PowerShell 7 and runs on Windows, Linux and macOS; Windows PowerShell 5.1 is refused up front via `#requires`. Merges a thin, non-overlapping feed set into one de-duplicated, bogon-filtered file. Parsing runs in a compiled `Add-Type` hot loop (~1s for ~4M lines instead of minutes). Written to a `.tmp` sibling and swapped with `File.Replace`, so the shard never reads a half-written list, and a total feed outage refuses to overwrite a good list with an empty one. Re-running is idempotent — it exits without downloading anything while the list on disk is younger than `-MinInterval` (default 2h, the anchor feed's own refresh period), so a misconfigured scheduler can't hammer upstream.

## CrowdSec: contribute-first

`IBanReporter` + `BanChannel` fan locally-decided bans out to external systems. `CrowdSecReporter` (UOContent) posts to LAPI `POST /v1/alerts` and retracts via `DELETE /v1/decisions`.

Reporting is **enqueue-only** on the accept path: a bounded, coalescing channel drained off-loop with bounded retry, counted drops on overflow, and a flush on shutdown. Under a DDoS the accept path never does synchronous or lock-contending per-IP work.

### Why not pull decisions from CrowdSec?

The original design streamed decisions into an in-app snapshot and enforced them at the accept gate. That's the wrong layer: by the time the shard sees the connection, the TCP handshake and socket setup are already paid for. `cs-firewall-bouncer` drops the same traffic **at the kernel**, and it's what CrowdSec is built to do. So the shard now contributes what it uniquely knows (rate-limit trips, blocklist hits from real connection attempts) and lets the OS enforce.

The one thing the OS can't do — hold millions of entries on Windows — is exactly what the in-app blocklist covers, and it feeds the same pipeline.

## Threading policy

`CLAUDE.md` rule #3 is rewritten as an explicit three-part policy, with rule #10 restated in tandem:

- Anything touching game state runs **only** on the main loop.
- Heavy work that *needs* game state must be **chunked** across ticks, never threaded.
- Heavy work that does *not* need game state (large-file parse, external I/O) **must** run off-loop **and must yield to world saves**.

Results come back via an immutable snapshot swapped through a single `volatile` reference, or `Core.LoopContext.Post` — never by letting the scheduler decide where heavy work runs. Both new subsystems follow it.

## Shared primitives

`SortedRangeIndex<T> where T : IBinaryInteger<T>` — coalesced disjoint interval arrays plus a binary search. The firewall, the blocklist, and (as of this PR) core's reserved-network tables all use it.

Coalescing is a correctness requirement, not an optimization: multi-feed lists nest CIDRs (`/24` containing a `/32`), and a search that inspects only the rightmost run whose minimum is ≤ the value is sound **only** over disjoint runs. That bug was caught in review and is covered by regression tests.

`IPAddressUtility` collects the allocation-free `IPAddress` ↔ `UInt128` conversions and CIDR parsing that were previously scattered or duplicated.

## Config

| File | Owner | Keys |
|---|---|---|
| `Configuration/bans.json` | core | `reportRateLimitTrips`, `autoBanDuration` |
| `Configuration/blocklist.json` | content | `file`, `reloadInterval`, `reportHits`, `banDuration`, `promoteSuppression` |
| `Configuration/crowdsec.json` | content | `lapiUrl`, `machineId`, `password`, `origin`, `manualBanDuration`, `flushInterval`, `maxQueue` |
| `Configuration/firewall.json` | content | persisted firewall entries (migrated from `firewall.cfg`) |

Everything is inert by default. CrowdSec self-disables without credentials; the blocklist self-disables until its file exists. A shard that changes nothing sees no behavior change.

## Notes for review

- **Core no longer references `Firewall` or `IFirewallEntry` anywhere.** `NetworkUtilities` used to build its reserved-network tables out of `CidrFirewallEntry`, which coupled core to the firewall for something unrelated to banning; those are now a `SortedRangeIndex<UInt128>`, same semantics and public API.
- **`BanChannel.Stop()` no longer persists the firewall** — a contribution coordinator has no business saving an enforcement store. That's the firewall filter's `Stop()`.
- **A dead `whitelisted` parameter was dropped** from the blocklist gate: it was hardcoded `false` at its only call site, and no whitelist concept exists in core.
- **The blocklist filter is an instance, not a static.** The static version forced its tests onto the sequential collection with a reset hook; they now run in parallel.
- `dev-docs/networking-packets.md` documents the seam for content authors, plus a known wart in the `IPAddress` ↔ `UInt128` normalization flagged for a follow-up PR.
- The generator was verified on Linux, macOS and Windows under a temporary CI matrix (since removed). It caught two portability bugs — a Windows-only path separator, and a culture-sensitive duration parse that read `2.5` as `25` on comma-decimal locales and *silently* turned a 2.5h cooldown into 25h — plus a third that made the script unparseable on Windows PowerShell 5.1. The source is ASCII-only for that last reason: `#requires` is only honored once a file parses, so non-ASCII in a BOM-less script produces parse errors instead of the version message.

## Tests

**1344 pass** (782 `Server.Tests`, 562 `UOContent.Tests`). New coverage: filter registry (registration, short-circuit, fault-disable), blocklist parsing/CIDR/coalescing, snapshot reload markers, promote-guard TTL, ban-channel fan-out, CrowdSec alert building/dedup/flush-on-stop, and the generator's output-format contract pinned against the reader.
2026-07-25 11:59:37 -07:00

669 lines
20 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: Main.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Server.Compression;
using Server.Json;
using Server.Logging;
using Server.Network;
using Server.Network.Bans;
using Server.Text;
namespace Server;
public static class Core
{
private static readonly ILogger logger = LogFactory.GetLogger(typeof(Core));
private static bool _performProcessKill;
private static bool _restartOnKill;
private static bool _performSnapshot;
private static string _snapshotPath;
private static bool _crashed;
private static string _baseDirectory;
private static bool? _isRunningFromXUnit;
private static int _itemCount;
private static int _mobileCount;
public static EventLoopContext LoopContext { get; set; }
private static readonly Type[] _serialTypeArray = { typeof(Serial) };
public static readonly bool IsWindows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
public static readonly bool IsDarwin = RuntimeInformation.IsOSPlatform(OSPlatform.OSX);
public static readonly bool IsFreeBSD = RuntimeInformation.IsOSPlatform(OSPlatform.FreeBSD);
public static readonly bool IsLinux = RuntimeInformation.IsOSPlatform(OSPlatform.Linux) || IsFreeBSD;
public static readonly bool IsBSD = IsDarwin || IsFreeBSD;
public static readonly bool Unix = IsBSD || IsLinux;
private const string AssembliesConfiguration = "Data/assemblies.json";
#nullable enable
// TODO: Find a way to get rid of this
public static bool IsRunningFromXUnit
{
get
{
if (_isRunningFromXUnit != null)
{
return _isRunningFromXUnit.Value;
}
foreach (var a in AppDomain.CurrentDomain.GetAssemblies())
{
if (a.FullName.InsensitiveStartsWith("xunit"))
{
_isRunningFromXUnit = true;
return true;
}
}
_isRunningFromXUnit = false;
return false;
}
}
#nullable restore
public static Assembly ApplicationAssembly { get; set; }
public static Assembly Assembly { get; set; }
// Assembly file version
public static Version Version => new(ThisAssembly.AssemblyFileVersion);
public static Process Process { get; private set; }
public static Thread Thread { get; private set; }
private static long _firstTick;
// Make these available to unit tests for mocking
internal static long _tickCount;
internal static DateTime _now;
public static long TickCount => _tickCount;
public static DateTime Now => _now;
public static long Uptime => TickCount - _firstTick;
private static double _currentCPS;
private static double _averageCPS;
private static bool _cpsInitialized;
public static double CyclesPerSecond => _currentCPS;
public static double AverageCPS => _averageCPS;
public static string BaseDirectory
{
get
{
if (_baseDirectory == null)
{
try
{
_baseDirectory = ApplicationAssembly.Location;
if (_baseDirectory.Length > 0)
{
_baseDirectory = Path.GetDirectoryName(_baseDirectory);
}
}
catch
{
_baseDirectory = "";
}
}
return _baseDirectory;
}
}
public static CancellationTokenSource ClosingTokenSource { get; } = new();
public static bool Closing => ClosingTokenSource.IsCancellationRequested;
public static bool Headless { get; private set; }
public static int GlobalUpdateRange { get; set; } = 18;
public static int GlobalMaxUpdateRange { get; set; } = 24;
public static int ScriptItems => _itemCount;
public static int ScriptMobiles => _mobileCount;
public static Expansion Expansion { get; set; }
public static bool T2A => Expansion >= Expansion.T2A;
public static bool UOR => Expansion >= Expansion.UOR;
public static bool UOTD => Expansion >= Expansion.UOTD;
public static bool LBR => Expansion >= Expansion.LBR;
public static bool AOS => Expansion >= Expansion.AOS;
public static bool SE => Expansion >= Expansion.SE;
public static bool ML => Expansion >= Expansion.ML;
public static bool SA => Expansion >= Expansion.SA;
public static bool HS => Expansion >= Expansion.HS;
public static bool TOL => Expansion >= Expansion.TOL;
public static bool EJ => Expansion >= Expansion.EJ;
public static string FindDataFile(string path, bool throwNotFound = true)
{
string fullPath = null;
foreach (var p in ServerConfiguration.DataDirectories)
{
fullPath = Path.Combine(p, path);
if (IsLinux && !File.Exists(fullPath))
{
var fi = new FileInfo(fullPath);
if (fi.Directory != null && Directory.Exists(fi.Directory.FullName))
{
fullPath = fi.Directory.EnumerateFiles(
fi.Name,
new EnumerationOptions { MatchCasing = MatchCasing.CaseInsensitive }
).FirstOrDefault()?.FullName;
}
}
if (File.Exists(fullPath))
{
break;
}
fullPath = null;
}
if (fullPath == null && throwNotFound)
{
throw new FileNotFoundException($"Data: {path} was not found");
}
return fullPath;
}
public static IEnumerable<string> FindDataFileByPattern(string pattern)
{
var options = new EnumerationOptions { MatchCasing = MatchCasing.CaseInsensitive };
foreach (var p in ServerConfiguration.DataDirectories)
{
if (Directory.Exists(p))
{
foreach (var file in Directory.EnumerateFiles(p, pattern, options))
{
yield return file;
}
}
}
}
public static void Kill(bool restart = false)
{
_restartOnKill = restart;
_performProcessKill = true;
}
public static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
Console.WriteLine(e.IsTerminating ? "Error:" : "Warning:");
Console.WriteLine(e.ExceptionObject);
if (e.IsTerminating)
{
_crashed = true;
var close = false;
try
{
var args = new ServerCrashedEventArgs(e.ExceptionObject as Exception);
EventSink.InvokeServerCrashed(args);
close = args.Close;
}
catch
{
// ignored
}
if (!close && !Headless)
{
Console.WriteLine("This exception is fatal, press return to exit");
ConsoleInputHandler.ReadLine();
}
DoKill();
}
}
private static void CurrentDomain_ProcessExit(object sender, EventArgs e)
{
if (!Closing)
{
HandleClosed();
}
}
private static void Console_CancelKeyPressed(object sender, ConsoleCancelEventArgs e)
{
var keypress = e.SpecialKey switch
{
ConsoleSpecialKey.ControlBreak => "CTRL+BREAK",
_ => "CTRL+C"
};
logger.Information("Detected {Key} pressed.", keypress);
e.Cancel = true;
Kill();
}
internal static void DoKill(bool restart = false)
{
if (Closing)
{
return;
}
HandleClosed();
if (restart)
{
try
{
logger.Information("Restarting");
if (IsWindows)
{
using var process = Process.Start("dotnet", $"{ApplicationAssembly.Location}");
}
else
{
using var process = new Process();
process.StartInfo = new ProcessStartInfo
{
FileName = "dotnet",
Arguments = $"{ApplicationAssembly.Location}",
UseShellExecute = true
};
process.Start();
}
logger.Information("Restart done");
}
catch (Exception e)
{
logger.Error(e, "Restart failed");
}
}
Environment.Exit(0);
}
private static void HandleClosed()
{
ClosingTokenSource.Cancel();
logger.Information("Shutting down");
World.WaitForWriteCompletion();
World.ExitSerializationThreads();
PingServer.Shutdown();
NetState.Shutdown();
BanChannel.Stop();
ConnectionFilters.Stop();
if (!_crashed)
{
EventSink.InvokeShutdown();
}
}
private static readonly bool UseFastTimestampMath = Stopwatch.Frequency % 1000 == 0;
private static readonly ulong FrequencyInMilliseconds = (ulong)Stopwatch.Frequency / 1000;
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static long GetTimestamp()
{
if (UseFastTimestampMath)
{
return (long)((ulong)Stopwatch.GetTimestamp() / FrequencyInMilliseconds);
}
// Fast calculation will be lossy, fallback to slower but accurate calculation
return (long)((UInt128)Stopwatch.GetTimestamp() * 1000 / (ulong)Stopwatch.Frequency);
}
public static void Setup(Assembly applicationAssembly, Process process)
{
CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture;
Process = process;
ApplicationAssembly = applicationAssembly;
Assembly = Assembly.GetAssembly(typeof(Core));
Thread = Thread.CurrentThread;
LoopContext = new EventLoopContext();
SynchronizationContext.SetSynchronizationContext(LoopContext);
AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException;
AppDomain.CurrentDomain.ProcessExit += CurrentDomain_ProcessExit;
AppDomain.CurrentDomain.AssemblyResolve += AssemblyHandler.AssemblyResolver;
Console.OutputEncoding = Encoding.UTF8;
Thread.Name = "Core Thread";
if (BaseDirectory.Length > 0)
{
Directory.SetCurrentDirectory(BaseDirectory);
}
Utility.PushColor(ConsoleColor.Green);
Console.WriteLine(
"ModernUO - [https://github.com/modernuo/modernuo] Version {0}.{1}.{2}.{3}",
Version.Major,
Version.Minor,
Version.Build,
Version.Revision
);
Utility.PopColor();
Utility.PushColor(ConsoleColor.DarkGray);
Console.WriteLine(@"Copyright 2019-2026 ModernUO Development Team
This program comes with ABSOLUTELY NO WARRANTY;
This is free software, and you are welcome to redistribute it under certain conditions.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
".TrimMultiline());
Utility.PopColor();
Console.CancelKeyPress += Console_CancelKeyPressed;
Headless = Console.IsInputRedirected;
if (Headless)
{
logger.Information("Headless mode detected (stdin is not a TTY); interactive console input is disabled.");
}
// LibDeflate is not thread safe, so we need to create a new instance for each thread
var standard = Deflate.Standard;
AppDomain.CurrentDomain.ProcessExit += (_, _) => standard.Dispose();
ServerConfiguration.Load();
var assemblyPath = Path.Join(BaseDirectory, AssembliesConfiguration);
// Load UOContent.dll
var assemblyFiles = JsonConfig.Deserialize<List<string>>(assemblyPath)?.ToArray();
if (assemblyFiles == null)
{
throw new JsonException($"Failed to deserialize {assemblyPath}.");
}
for (var i = 0; i < assemblyFiles.Length; i++)
{
assemblyFiles[i] = Path.Join(BaseDirectory, "Assemblies", assemblyFiles[i]);
}
AssemblyHandler.LoadAssemblies(assemblyFiles);
// First-boot interactive setup. Runs after assemblies are loaded (so content can
// register prompts) but before any Serilog output, so console prompts are not
// interleaved with the async console sink. Handlers self-gate on first-boot state
// (e.g. "is my setting already present?").
AssemblyHandler.Invoke("ConfigurePrompts");
logger.Information("Running on {Framework}", RuntimeInformation.FrameworkDescription);
VerifySerialization();
_now = DateTime.UtcNow;
_firstTick = _tickCount = GetTimestamp();
Timer.Init(_tickCount);
AssemblyHandler.Invoke("Configure");
TileMatrixLoader.LoadTileMatrix();
RegionJsonSerializer.LoadRegions();
World.Load();
AssemblyHandler.Invoke("Initialize");
BanChannel.Start(ClosingTokenSource.Token);
ConnectionFilters.Start(ClosingTokenSource.Token);
NetState.Start();
PingServer.Start();
EventSink.InvokeServerStarted();
RunEventLoop();
}
public static void RunEventLoop()
{
try
{
var lastRaw = Stopwatch.GetTimestamp();
const int interval = 100;
double frequency = Stopwatch.Frequency * interval;
const double alpha = 2.0 / 129; // EMA smoothing (≈128-sample window)
var sample = 0;
while (!Closing)
{
_tickCount = GetTimestamp();
_now = DateTime.UtcNow;
Mobile.ProcessDeltaQueue();
Item.ProcessDeltaQueue();
Timer.Slice(_tickCount);
// Handle networking
NetState.Slice();
// Execute captured post-await methods (like Timer.Pause)
LoopContext.ExecuteTasks();
Timer.CheckTimerPool(); // Check for pool depletion so we can async refill it.
if (_performSnapshot)
{
// Return value is the offset that can be used to fix timers that should drift
World.Snapshot(_snapshotPath);
_performSnapshot = false;
}
if (_performProcessKill)
{
World.WaitForWriteCompletion();
break;
}
if (sample++ == interval)
{
sample = 0;
var nowRaw = Stopwatch.GetTimestamp();
_currentCPS = frequency / (nowRaw - lastRaw);
if (!_cpsInitialized)
{
_averageCPS = _currentCPS;
_cpsInitialized = true;
}
else
{
_averageCPS += alpha * (_currentCPS - _averageCPS);
}
lastRaw = nowRaw;
var sleepMs = (int)Timer.MillisecondsUntilNextTick(_tickCount);
if (sleepMs >= 2)
{
NetState.WaitForCompletion(sleepMs - 1);
}
}
}
}
catch (Exception e)
{
CurrentDomain_UnhandledException(null, new UnhandledExceptionEventArgs(e, true));
return;
}
DoKill(_restartOnKill);
}
internal static void RequestSnapshot(string snapshotPath)
{
_snapshotPath = snapshotPath;
_performSnapshot = true;
}
public static void VerifySerialization()
{
_itemCount = 0;
_mobileCount = 0;
var callingAssembly = Assembly.GetCallingAssembly();
VerifySerialization(callingAssembly);
foreach (var assembly in AssemblyHandler.Assemblies)
{
if (assembly != callingAssembly)
{
VerifySerialization(assembly);
}
}
}
private static void VerifyType(Type type)
{
if (!type.IsAssignableTo(typeof(ISerializable)) || type.IsInterface || type.IsAbstract)
{
return;
}
if (type.IsSubclassOf(typeof(Item)))
{
Interlocked.Increment(ref _itemCount);
}
else if (type.IsSubclassOf(typeof(Mobile)))
{
Interlocked.Increment(ref _mobileCount);
}
using var errors = ValueStringBuilder.CreateMT();
try
{
if (World.DirtyTrackingEnabled)
{
var manualDirtyCheckingAttribute = type.GetCustomAttribute<ManualDirtyCheckingAttribute>(false);
var codeGennedAttribute = type.GetCustomAttribute<ModernUO.Serialization.SerializationGeneratorAttribute>(false);
if (manualDirtyCheckingAttribute == null && codeGennedAttribute == null)
{
errors.AppendLine(" - No property tracking (dirty checking)");
}
}
if (type.GetConstructor(_serialTypeArray) == null)
{
errors.AppendLine(" - No serialization constructor");
}
const BindingFlags bindingFlags = BindingFlags.Public | BindingFlags.NonPublic |
BindingFlags.Instance | BindingFlags.DeclaredOnly;
var hasSerializeMethod = false;
var hasDeserializeMethod = false;
foreach (var method in type.GetMethods(bindingFlags))
{
if (method.Name == "Serialize")
{
hasSerializeMethod = true;
}
if (method.Name == "Deserialize")
{
var parameters = method.GetParameters();
if (parameters.Length == 1 && parameters[0].ParameterType == typeof(IGenericReader))
{
hasDeserializeMethod = true;
}
}
}
if (!hasSerializeMethod)
{
errors.AppendLine(" - No Serialize() method");
}
if (!hasDeserializeMethod)
{
errors.AppendLine(" - No Deserialize() method");
}
if (errors.Length > 0)
{
Utility.PushColor(ConsoleColor.Red);
Console.WriteLine($"{type}{Environment.NewLine}{errors.ToString()}");
Utility.PopColor();
}
}
catch (AmbiguousMatchException e)
{
// ignored
}
catch
{
Console.WriteLine("Warning: Exception in serialization verification of type {0}", type);
}
}
private static void VerifySerialization(Assembly assembly)
{
if (assembly != null)
{
Parallel.ForEach(assembly.GetTypes(), VerifyType);
}
}
}