ModernUO/Projects/UOContent/Network/Blocklist/BlocklistSnapshot.cs
Kamron Batman 927c87702d
feat(network): allowlist false-positive IPs, escalate on behaviour
The shard owner, on a Starlink CGNAT address, was blocked by the imported
reputation blocklist. The cause was not CrowdSec: the address was a literal line
in ip-blocklist.txt, so BlocklistFilter denied it at accept and then promoted it.
Clearing the CrowdSec decision could not fix it either, because the file entry
re-reports within promoteSuppression of every reconnect.

Reputation feeds list shared consumer address space constantly. On CGNAT one
public address fronts many subscribers at the same time, so a single abusive
customer gets the address listed and everyone else behind it is blocked with
them; where leases rotate, a listing says little about whoever holds the address
now. So exemptions go where they cost nothing, and escalation is driven by what a
connection actually does.

GENERATOR (tools/Export-IpBlocklist.ps1)

- -AllowlistFile takes multiple paths, subtracted from the merged set before the
  output is written. Defaults to the operator's own ip-allowlist.txt (created
  once, never rewritten) plus one generated file per network carve-out.
- Subtraction is range-correct: an allowlisted address inside a blocked CIDR
  splits that CIDR around the hole instead of being ignored. This also fixes
  -ExcludeAnonymizers, which parsed CIDR entries and then only subtracted
  singles.
- Carve-outs are a table (name, ASN, reason, offline seed) rather than a
  hardcoded network, so covering another CGNAT provider is one row.
  -RefreshCarveouts re-fetches from the ASN's current routing announcements and
  collapses them; -Carveout '' subtracts none. Announcements rather than
  ownership records, because registry data disagrees with what is routed and caps
  its result sets.
- Editing an allowlist bypasses -MinInterval, so a just-added exemption is not
  indistinguishable from the allowlist not working.
- The shipped starlink carve-out costs ~0.1% of the list.

ALLOWLISTS

- FileAllowlist reads the same files the generator subtracts, so an operator
  entry means "leave this address alone" for real. Subtraction alone only covers
  being BLOCKED; behavioural detections never consult the blocklist, so without
  this a carve-out was quietly routed around and one scanner behind a shared
  address was enough to get everyone behind it firewalled. Reading the files also
  means an entry applies on the next reload rather than the next regeneration.
- LoginAllowlist is earned by authenticating, with a 90 day TTL because an
  address that logged in years ago is a stranger. Its own store rather than
  Account.LoginIPs, which has no timestamps and cannot be backfilled. An entry is
  evidence rather than a licence: 10 suppressed contributions in an hour revokes
  it, and a fresh login forgives the tally.
- Both are consulted only AFTER the blocklist has already matched, so a normal
  accept pays nothing for them and the accept gate stays allowlist-free.
- BanExemptions combines them behind BanChannel.IsExempt. Suppresses escalation
  only; every local defence still applies.

BEHAVIOURAL DETECTION

- silent-connect (reaped having sent zero bytes) and invalid-seed (a zero seed)
  are contributed. Both were already disconnected.
- ForeignProtocol positively identifies HTTP, TLS and SSH. Asking "is this a good
  UO client?" cannot work: LoginEncryption.ClientDecrypt is a byte-for-byte
  stream XOR, so a client with encryption on when the shard expects none sends a
  structurally perfect connection whose payload is noise. Nothing assumes arrival
  framing, since TCP has no message boundaries and a rule of the form "these
  bytes must arrive together" drops real players on poor links.
- Keyed on bytes-received rather than elapsed time throughout. A connection that
  sent something and ran out of time is far more likely a slow link, and banning
  those makes the player retry, trip the rate limiter, and compound it.
- AutoDenylist holds behavioural detections locally for 15 minutes, as an
  IConnectionFilter plus IBanReporter over one store so engine detection sites
  never reach into content. Closes the gap where a flood pays for a socket and a
  NetState per connection while waiting for an OS bouncer, and is the whole
  defence on a shard running none. Not persisted: a holding pen that survives
  restarts is a ban without a ban's review.
- BanReasons centralises the slugs. IsBehavioral is an opt-in set, not
  "everything except manual", so a future reason escalates normally instead of
  silently inheriting an exemption. The first cut of the exemption swallowed
  manual admin bans; this is why.

FIXES

- BanConfiguration.Settings was null until Configure() ran while the reap path
  dereferences it every Slice(), so a harness driving NetState.Slice() directly
  hit an NRE that looked flaky because it depended on test ordering.
- -AllowlistFile was typed [string] while documented and used as a list.

LAYOUT AND DOCS

Content network code moves out of Misc into UOContent/Network, one concern per
folder. Namespaces are untouched, so these are pure file moves.
dev-docs/ip-bans-and-allowlists.md documents the subsystem, leading with the
operator process for unblocking a player -- including the three things that look
sufficient and are not: deleting the CrowdSec decision alone, editing
ip-blocklist.txt by hand, and cscli allowlists alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 18:31:34 -07:00

211 lines
7.8 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: BlocklistSnapshot.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Buffers.Text;
using System.Net;
using System.Net.Sockets;
using System.Text;
using Server.Collections;
namespace Server.Network.Bans;
/// <summary>
/// Immutable dual-stack blocklist. Singles and CIDRs are folded into a single sorted, coalesced
/// interval index per family: IPv4 as <see cref="uint"/> ranges (lean for the millions-strong common
/// case), IPv6 as <see cref="UInt128"/> ranges (empty unless the feed carries v6). Immutable → lock-free reads.
/// </summary>
public sealed class BlocklistSnapshot
{
public static readonly BlocklistSnapshot Empty = new(SortedRangeIndex<uint>.Empty, SortedRangeIndex<UInt128>.Empty);
private readonly SortedRangeIndex<uint> _v4;
private readonly SortedRangeIndex<UInt128> _v6;
public int Count => _v4.Count + _v6.Count;
private BlocklistSnapshot(SortedRangeIndex<uint> v4, SortedRangeIndex<UInt128> v6)
{
_v4 = v4;
_v6 = v6;
}
/// <summary>
/// Parses a blocklist directly from its UTF-8/ASCII file bytes — one line at a time, splitting on
/// <c>'\n'</c> with no per-line string allocation. IPv4 singles and CIDRs are parsed straight from the
/// byte span; IPv6 (the rare path) decodes the single address token and defers to the framework parser.
/// Malformed lines increment <paramref name="skipped"/> and never throw. Build-time intermediates use
/// the multithreaded pool because this runs off the game loop on the reload/bootstrap thread.
/// </summary>
public static BlocklistSnapshot Build(ReadOnlySpan<byte> data, out int parsed, out int skipped)
{
parsed = 0;
skipped = 0;
// Only the two final index arrays (allocated inside SortedRangeIndex.Build) hit the heap; every
// build-time buffer here is a pooled ref list. mt: true is required — this runs off the game loop.
using var v4 = PooledRefList<SortedRangeIndex<uint>.Range>.Create(mt: true);
using var v6 = PooledRefList<SortedRangeIndex<UInt128>.Range>.Create(mt: true);
var rest = data;
while (!rest.IsEmpty)
{
ReadOnlySpan<byte> line;
var nl = rest.IndexOf((byte)'\n');
if (nl >= 0)
{
line = rest[..nl];
rest = rest[(nl + 1)..];
}
else
{
line = rest;
rest = default;
}
line = line[Ascii.Trim(line)];
if (line.IsEmpty || line[0] == (byte)'#' || line[0] == (byte)';')
{
continue;
}
var slash = line.IndexOf((byte)'/');
var addr = slash >= 0 ? line[..slash] : line;
var bitsToken = slash >= 0 ? line[(slash + 1)..] : default;
if (addr.IndexOf((byte)':') < 0)
{
// IPv4 single or CIDR — parsed straight from the byte span.
if (slash >= 0)
{
if (IPAddressUtility.TryParseV4(addr, out var ip) &&
TryParseBits(bitsToken, out var bits) && bits is >= 0 and <= 32)
{
var size = bits == 0 ? 0xFFFFFFFFu : (1u << (32 - bits)) - 1;
var b = ip & ~size;
v4.Add(new SortedRangeIndex<uint>.Range(b, b + size));
parsed++;
}
else
{
skipped++;
}
}
else if (IPAddressUtility.TryParseV4(addr, out var ip))
{
v4.Add(new SortedRangeIndex<uint>.Range(ip, ip));
parsed++;
}
else
{
skipped++;
}
}
else if (TryDecodeV6(addr, out var v))
{
// IPv6 is rare in these feeds; the single token was decoded and framework-parsed above.
if (slash >= 0)
{
if (TryParseBits(bitsToken, out var bits) && bits is >= 0 and <= 128)
{
var mask = bits == 0 ? UInt128.Zero : ~((UInt128.One << (128 - bits)) - 1);
var b = v & mask;
v6.Add(new SortedRangeIndex<UInt128>.Range(b, b | ~mask));
parsed++;
}
else
{
skipped++;
}
}
else
{
v6.Add(new SortedRangeIndex<UInt128>.Range(v, v));
parsed++;
}
}
else
{
skipped++;
}
}
v4.Sort(SortedRangeIndex<uint>.ByMin);
v6.Sort(SortedRangeIndex<UInt128>.ByMin);
return new BlocklistSnapshot(SortedRangeIndex<uint>.Build(v4.AsSpan()), SortedRangeIndex<UInt128>.Build(v6.AsSpan()));
}
// Decodes a single IPv6 address token from ASCII bytes and validates it via the framework parser.
private static bool TryDecodeV6(ReadOnlySpan<byte> addr, out UInt128 v)
{
v = UInt128.Zero;
if (addr.Length > 45)
{
return false;
}
Span<char> chars = stackalloc char[addr.Length];
for (var i = 0; i < addr.Length; i++)
{
chars[i] = (char)addr[i];
}
if (!IPAddress.TryParse(chars, out var a) || a.AddressFamily != AddressFamily.InterNetworkV6)
{
return false;
}
v = a.ToUInt128();
return true;
}
private static bool TryParseBits(ReadOnlySpan<byte> token, out int bits)
{
if (Utf8Parser.TryParse(token, out bits, out var consumed) && consumed == token.Length)
{
return true;
}
bits = 0;
return false;
}
/// <summary>
/// Plain set membership, for callers whose set is an ALLOWlist (see <see cref="FileAllowlist"/>) and for
/// whom <see cref="IsBanned"/> would read backwards. The interval machinery is direction-agnostic.
/// </summary>
public bool Contains(IPAddress ip) => IsBanned(ip);
public bool IsBanned(IPAddress ip)
{
if (ip.IsIPv4MappedToIPv6)
{
// v6-encoded v4 must not dodge the v4 set; extract the embedded v4 uint directly.
return IPAddressUtility.TryMappedV4(ip, out var mv) && _v4.Contains(mv);
}
if (ip.AddressFamily == AddressFamily.InterNetwork)
{
return IPAddressUtility.TryV4(ip, out var v) && _v4.Contains(v);
}
if (ip.AddressFamily == AddressFamily.InterNetworkV6)
{
return _v6.Contains(ip.ToUInt128());
}
return false;
}
}