ModernUO/Projects/UOContent/Network/Packets/IncomingAccountPackets.cs
Kamron Batman a7e65aab01
perf(login): run password hashing on a parked worker thread (#2566)
## Why

An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.

## What the measurement says

Off-loading does not delete the cost, it relocates it. Three things stay on the loop:

| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |

Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.

Two results shaped the design:

- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.

## Why exactly one worker

It is load-bearing three times over, which is also why it must not quietly become a pool:

- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.

Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.

## Making every protection safe off-thread

The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:

| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |

Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.

With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".

## Correctness

- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.

## Bounding

`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.

The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.

## Gate

Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.

## Engine change

One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.

## Docs

`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.

## Testing

698 UOContent tests, 810 Server tests, Release build clean.

Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
2026-08-09 00:13:34 -07:00

649 lines
21 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: IncomingAccountPackets.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Buffers;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Security.Cryptography;
using Server.Accounting;
using Server.Engines.CharacterCreation;
using Server.Misc;
using Server.Mobiles;
namespace Server.Network;
public static class IncomingAccountPackets
{
// Initial capacity and the point at which issuing sweeps expired ids. Not a cap; the window
// grows rather than evicting a live id.
private const int _authIDWindowSize = 128;
private static int _authIdPurgeThreshold = _authIDWindowSize;
// The gap between PlayServerAck and the game login is seconds. Bounds how long a stolen id
// stays usable.
private static readonly TimeSpan _authIDLifetime = TimeSpan.FromMinutes(2.0);
private static readonly Dictionary<int, AuthIDPersistence> _authIDWindow =
new(_authIDWindowSize);
internal struct AuthIDPersistence
{
public DateTime Age;
public readonly ClientVersion Version;
// GameLogin skips its password verify when both match, so the id is a bearer token and has
// to be bound to whatever earned it.
public readonly IAccount Account;
public readonly IPAddress Address;
public AuthIDPersistence(ClientVersion v, IAccount account, IPAddress address)
{
Age = Core.Now;
Version = v;
Account = account;
Address = Utility.Intern(address);
}
}
internal enum AuthIdResult
{
// No such id, or it was issued for a different account or address.
Rejected,
// Right account and address, too old to stand in for the verify. Idling on the server list
// is normal, so this falls back to the password check rather than becoming a lockout.
Expired,
// Issued to this account, from this address, recently. Stands in for the password verify.
Vouched
}
public static unsafe void Configure()
{
IncomingPackets.Register(0x00, &CreateCharacter, 104, outgameOnly: true);
IncomingPackets.Register(0x5D, &PlayCharacter, 73, outgameOnly: true);
IncomingPackets.Register(0x80, &AccountLogin, 62, outgameOnly: true);
IncomingPackets.Register(0x83, &DeleteCharacter, 39, outgameOnly: true);
IncomingPackets.Register(0x91, &GameLogin, 65, outgameOnly: true);
IncomingPackets.Register(0xA0, &PlayServer, 3, outgameOnly: true);
IncomingPackets.Register(0xBD, &ClientVersion);
IncomingPackets.Register(0xE1, &ClientType);
IncomingPackets.Register(0xEF, &LoginServerSeed, 21, outgameOnly: true);
IncomingPackets.Register(0xF8, &CreateCharacter, 106, outgameOnly: true);
}
public static void CreateCharacter(NetState state, SpanReader reader)
{
reader.Seek(9, SeekOrigin.Current);
/*
var unk1 = reader.ReadInt32();
var unk2 = reader.ReadInt32();
int unk3 = reader.ReadByte();
*/
var name = reader.ReadLatin1Safe(30);
reader.Seek(2, SeekOrigin.Current);
var flags = reader.ReadInt32();
reader.Seek(8, SeekOrigin.Current);
int prof = reader.ReadByte();
reader.Seek(15, SeekOrigin.Current);
var genderRace = reader.ReadByte();
// Strength, Dex, Intelligence
byte[] stats = [reader.ReadByte(), reader.ReadByte(), reader.ReadByte()];
var skills = new (SkillName, byte)[state.NewCharacterCreation ? 4 : 3];
skills[0] = ((SkillName)reader.ReadByte(), reader.ReadByte());
skills[1] = ((SkillName)reader.ReadByte(), reader.ReadByte());
skills[2] = ((SkillName)reader.ReadByte(), reader.ReadByte());
if (state.NewCharacterCreation)
{
skills[3] = ((SkillName)reader.ReadByte(), reader.ReadByte());
}
int hue = reader.ReadUInt16();
int hairVal = reader.ReadInt16();
int hairHue = reader.ReadInt16();
int hairValf = reader.ReadInt16();
int hairHuef = reader.ReadInt16();
reader.ReadByte();
int cityIndex = reader.ReadByte();
reader.Seek(8, SeekOrigin.Current);
/*
var charSlot = reader.ReadInt32();
var clientIP = reader.ReadInt32();
*/
int shirtHue = reader.ReadInt16();
int pantsHue = reader.ReadInt16();
/*
Pre-7.0.0.0:
0x00, 0x01 -> Human Male, Human Female
0x02, 0x03 -> Elf Male, Elf Female
Post-7.0.0.0:
0x00, 0x01
0x02, 0x03 -> Human Male, Human Female
0x04, 0x05 -> Elf Male, Elf Female
0x05, 0x06 -> Gargoyle Male, Gargoyle Female
*/
var female = genderRace % 2 != 0;
var raceID = state.StygianAbyss ? (byte)(genderRace < 4 ? 0 : genderRace / 2 - 1) : (byte)(genderRace / 2);
var race = Race.Races[raceID] ?? Race.DefaultRace;
var info = state.CityInfo;
var a = state.Account;
if (info == null || a == null || cityIndex >= info.Length)
{
state.Disconnect("Invalid city selected during character creation.");
return;
}
// Check if anyone is using this account
for (var i = 0; i < a.Length; ++i)
{
var check = a[i];
if (check != null && check.Map != Map.Internal)
{
state.LogInfo("Account in use");
state.SendPopupMessage(PMMessage.CharInWorld);
return;
}
}
state.Flags = (ClientFlags)flags;
var args = new CharacterCreatedEventArgs(
state,
a,
name,
female,
hue,
stats,
info[cityIndex],
skills,
shirtHue,
pantsHue,
hairVal,
hairHue,
hairValf,
hairHuef,
prof,
race
);
state.SendClientVersionRequest();
state.BlockAllPackets = true;
CharacterCreation.CharacterCreatedEvent(args);
var m = args.Mobile;
if (m != null)
{
state.Mobile = m;
m.NetState = state;
new LoginTimer(state, m).Start();
}
else
{
state.BlockAllPackets = false;
state.Disconnect("Character creation blocked.");
}
}
public static void DeleteCharacter(NetState state, SpanReader reader)
{
reader.Seek(30, SeekOrigin.Current);
var index = reader.ReadInt32();
AccountHandler.DeleteRequest(state, index);
}
public static void ClientVersion(NetState state, SpanReader reader)
{
var version = state.Version = new ClientVersion(reader.ReadAscii());
// Record RTT if this is a response to our probe
state.RecordRttMeasurement();
ClientVerification.ClientVersionReceived(state, version);
}
public static void ClientType(NetState state, SpanReader reader)
{
reader.ReadUInt16();
int type = reader.ReadUInt16();
var version = state.Version = new ClientVersion(reader.ReadAscii());
// Record RTT if this is a response to our probe
state.RecordRttMeasurement();
ClientVerification.ClientVersionReceived(state, version);
}
public static void PlayCharacter(NetState state, SpanReader reader)
{
reader.Seek(36, SeekOrigin.Current); // 4 = 0xEDEDEDED, 30 = Name, 2 = unknown
var flags = reader.ReadInt32();
reader.Seek(24, SeekOrigin.Current);
var charSlot = reader.ReadInt32();
reader.Seek(4, SeekOrigin.Current); // var clientIP = reader.ReadInt32();
var a = state.Account;
if (a == null || charSlot < 0 || charSlot >= a.Length)
{
state.Disconnect("Invalid character slot selected.");
return;
}
var m = a[charSlot];
// Check if anyone is using this account
for (var i = 0; i < a.Length; ++i)
{
var check = a[i];
if (check != null && check.Map != Map.Internal && check != m)
{
state.LogInfo("Account in use");
state.SendPopupMessage(PMMessage.CharInWorld);
return;
}
}
if (m == null)
{
state.Disconnect("Empty character slot selected.");
return;
}
m.NetState?.Disconnect("Character selected for a player already logged in.");
state.SendClientVersionRequest();
state.BlockAllPackets = true;
state.Flags = (ClientFlags)flags;
state.Mobile = m;
m.NetState = state;
new LoginTimer(state, m).Start();
}
public static void DoLogin(this NetState state, Mobile m)
{
state.SendLoginConfirmation(m);
state.SendMapChange(m.Map);
state.SendMapPatches();
state.SendSeasonChange((byte)m.GetSeason(), true);
state.SendSupportedFeature();
state.ResetMovementState();
state.SendMobileUpdate(m);
state.SendMobileUpdate(m);
m.CheckLightLevels(true);
state.SendMobileUpdate(m);
state.SendMobileIncoming(m, m);
state.SendMobileStatus(m);
state.SendSetWarMode(m.Warmode);
m.SendEverything();
state.SendSupportedFeature();
state.SendMobileUpdate(m);
state.SendMobileStatus(m);
state.SendSetWarMode(m.Warmode);
state.SendMobileIncoming(m, m);
state.SendLoginComplete();
state.SendCurrentTime();
state.SendSeasonChange((byte)m.GetSeason(), true);
state.SendMapChange(m.Map);
state.SendPlayMusic(m.Region.Music);
if (m is PlayerMobile pm)
{
PlayerMobile.PlayerLoginEvent(pm);
}
}
private static int GenerateAuthID(this NetState state) =>
EnsureAuthId(state.AuthId, state.Account, state.Address, state.Version);
/// <summary>
/// One id per connection, by construction. Choosing a server queues a disconnect that is not
/// drained until the next slice, so a client pipelining another select into the same buffer
/// arrives here again; handing back the id it already holds cannot orphan one.
/// </summary>
internal static int EnsureAuthId(int existingAuthId, IAccount account, IPAddress address, ClientVersion version)
=> existingAuthId != 0 ? existingAuthId : RegisterAuthId(account, address, version);
internal static int RegisterAuthId(IAccount account, IPAddress address, ClientVersion version)
{
// Sweep the ids left behind by clients that picked a server and never arrived, but never
// evict a live one to make room -- the client holding it is on its way to redeem it. If all
// are live the window grows, which is a login rush, not a backlog. Each entry costs a
// successful password verify, so the size is self-limiting.
if (_authIDWindow.Count >= _authIdPurgeThreshold)
{
PurgeExpiredAuthIds();
_authIdPurgeThreshold = Math.Max(_authIDWindowSize, _authIDWindow.Count * 2);
}
int authID;
// The id stands in for a password verify, so it has to be unguessable. Zero is reserved:
// GameLogin reads state.AuthId == 0 as "no auth id was issued".
do
{
authID = RandomNumberGenerator.GetInt32(int.MinValue, int.MaxValue);
} while (authID == 0 || _authIDWindow.ContainsKey(authID));
_authIDWindow[authID] = new AuthIDPersistence(version, account, address);
return authID;
}
/// <summary>
/// Spends an auth id, but only for the account and address it was issued to. An address
/// mismatch is <see cref="AuthIdResult.Rejected"/> rather than a fallback: network switching
/// mid-login is not supported.
/// </summary>
internal static AuthIdResult ConsumeAuthId(int authId, string username, IPAddress address, out AuthIDPersistence entry)
{
if (!_authIDWindow.TryGetValue(authId, out entry))
{
return AuthIdResult.Rejected;
}
// Look, then take: removing before ownership is proven would let anyone landing on a live id
// burn it, leaving its owner to log in again. Address before username, so a remote guesser
// never learns whether a username matched.
if (!Utility.Intern(address).Equals(entry.Address)
|| entry.Account == null || !username.InsensitiveEquals(entry.Account.Username))
{
entry = default;
return AuthIdResult.Rejected;
}
// Theirs, so spend it. Expired counts as spent; it has done all it is ever going to do.
_authIDWindow.Remove(authId);
return Core.Now - entry.Age > _authIDLifetime ? AuthIdResult.Expired : AuthIdResult.Vouched;
}
private static void PurgeExpiredAuthIds()
{
var now = Core.Now;
foreach (var (key, entry) in _authIDWindow)
{
if (now - entry.Age > _authIDLifetime)
{
_authIDWindow.Remove(key);
}
}
}
internal static void ClearAuthIdWindow()
{
_authIDWindow.Clear();
_authIdPurgeThreshold = _authIDWindowSize;
}
internal static int AuthIdWindowCount => _authIDWindow.Count;
public static void GameLogin(NetState state, SpanReader reader)
{
if (state.SentFirstPacket)
{
state.Disconnect("Duplicate game login packet received.");
return;
}
state.SentFirstPacket = true;
var authId = reader.ReadInt32();
if (state.AuthId != 0 && authId != state.AuthId || state.AuthId == 0 && authId != state.Seed)
{
state.LogInfo("Invalid client detected, disconnecting...");
state.Disconnect("Invalid auth id in game login packet.");
return;
}
var username = reader.ReadLatin1Safe(30);
var password = reader.ReadLatin1Safe(30);
var authResult = ConsumeAuthId(authId, username, state.Address, out var ap);
if (authResult == AuthIdResult.Rejected)
{
state.LogInfo("Invalid client detected, disconnecting...");
state.Disconnect("Unable to find auth id.");
return;
}
state.Version = ap.Version;
state.Seeded = true;
// Expired carries a usable entry; only the password verify skip is withheld.
var e = new GameServer.GameLoginEventArgs(
state,
username,
password,
authResult == AuthIdResult.Vouched
);
GameServer.GameServerLoginEvent(e);
if (e.Accepted)
{
state.CityInfo = e.CityInfo;
// Comment out these lines to turn off huffman compression
state.CompressionEnabled = true;
state.SendSupportedFeature();
state.SendCharacterList();
}
else
{
state.Disconnect("Login rejected by GameLogin packet handler.");
}
}
public static void PlayServer(NetState state, SpanReader reader)
{
// A server is picked once per connection. Picking again hands back an id this connection may
// already have spent on a game login, which the client could never redeem.
if (state.AuthId != 0)
{
state.Disconnect("Duplicate play server packet sent.");
return;
}
int index = reader.ReadInt16();
var info = state.ServerInfo;
var a = state.Account;
if (info == null || a == null || index < 0 || index >= info.Length)
{
state.Disconnect("Invalid server selected.");
}
else
{
var si = info[index];
state.AuthId = state.GenerateAuthID();
state.SentFirstPacket = false;
state.SendPlayServerAck(si, state.AuthId);
}
}
public static void LoginServerSeed(NetState state, SpanReader reader)
{
// Seeding happens once per connection. A second one restarts a handshake this connection
// already completed, which no real client does.
if (state.Seeded)
{
state.Disconnect("Duplicate login server seed packet sent.");
return;
}
state.Seed = reader.ReadInt32();
state.Seeded = true;
if (state.Seed == 0)
{
state.LogInfo("Invalid client detected, disconnecting");
state.Disconnect("Invalid client detected");
return;
}
var clientMaj = reader.ReadInt32();
var clientMin = reader.ReadInt32();
var clientRev = reader.ReadInt32();
var clientPat = reader.ReadInt32();
state.Version = new ClientVersion(clientMaj, clientMin, clientRev, clientPat);
}
public static void AccountLogin(NetState state, SpanReader reader)
{
if (state.SentFirstPacket)
{
state.Disconnect("Duplicate account login packet sent.");
return;
}
state.SentFirstPacket = true;
var username = reader.ReadLatin1Safe(30);
var password = reader.ReadLatin1Safe(30);
var accountLoginEventArgs = new AccountLoginEventArgs(state, username, password);
EventSink.InvokeAccountLogin(accountLoginEventArgs);
// The password check moved off the loop; whoever took it replies when the verdict lands.
if (accountLoginEventArgs.Deferred)
{
return;
}
CompleteAccountLogin(state, accountLoginEventArgs.Accepted, accountLoginEventArgs.RejectReason);
}
/// <summary>
/// Replies to an account login. Split out so a verdict produced off the loop reaches the client
/// through exactly the same path as one produced inline.
/// </summary>
internal static void CompleteAccountLogin(NetState state, bool accepted, ALRReason rejectReason)
{
if (accepted)
{
var serverListEventArgs = new GatewayServer.ServerListEventArgs(state, state.Account);
GatewayServer.ServerListEvent(serverListEventArgs);
if (serverListEventArgs.Rejected)
{
state.Account = null;
AccountLogin_ReplyRej(state, ALRReason.BadComm);
}
else
{
state.ServerInfo = serverListEventArgs.Servers.ToArray();
state.SendAccountLoginAck();
}
}
else
{
state.Account = null;
AccountLogin_ReplyRej(state, rejectReason);
}
}
private static void AccountLogin_ReplyRej(this NetState state, ALRReason reason)
{
state.SendAccountLoginRejected(reason);
state.Disconnect($"Account login rejected due to {reason}");
}
private class LoginTimer : Timer
{
private readonly Mobile _mobile;
private readonly NetState _state;
public LoginTimer(NetState state, Mobile m) : base(TimeSpan.FromMilliseconds(64), TimeSpan.FromMilliseconds(64))
{
_state = state;
_mobile = m;
}
protected override void OnTick()
{
if (_state != null)
{
if (_state.Account == null)
{
_state.Disconnect("Account was deleted during the login process.");
}
else if (_mobile == null)
{
_state.Disconnect("Player was deleted during the login process.");
}
else if (_state.Version != null)
{
_state.BlockAllPackets = false;
DoLogin(_state, _mobile);
}
else // Waiting to receive the client version before we continue the login process
{
return;
}
}
Stop();
}
}
}