ModernUO/Projects/UOContent/Accounting/AccountHandler.cs
Kamron Batman a7e65aab01
perf(login): run password hashing on a parked worker thread (#2566)
## Why

An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.

## What the measurement says

Off-loading does not delete the cost, it relocates it. Three things stay on the loop:

| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |

Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.

Two results shaped the design:

- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.

## Why exactly one worker

It is load-bearing three times over, which is also why it must not quietly become a pool:

- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.

Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.

## Making every protection safe off-thread

The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:

| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |

Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.

With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".

## Correctness

- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.

## Bounding

`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.

The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.

## Gate

Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.

## Engine change

One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.

## Docs

`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.

## Testing

698 UOContent tests, 810 Server tests, Release build clean.

Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
2026-08-09 00:13:34 -07:00

497 lines
16 KiB
C#

using System;
using System.Buffers;
using System.Collections.Generic;
using System.Net;
using System.Runtime.CompilerServices;
using ModernUO.CodeGeneratedEvents;
using Server.Accounting;
using Server.Accounting.Security;
using Server.Engines.CharacterCreation;
using Server.Engines.Help;
using Server.Logging;
using Server.Network;
using Server.Regions;
namespace Server.Misc;
public static class AccountHandler
{
private static readonly ILogger logger = LogFactory.GetLogger(typeof(AccountHandler));
private static int MaxAccountsPerIP;
private static bool AutoAccountCreation;
private static readonly bool RestrictDeletion = !TestCenter.Enabled;
private static readonly TimeSpan DeleteDelay = TimeSpan.FromDays(7.0);
private static bool PasswordCommandEnabled;
private static Dictionary<IPAddress, int> m_IPTable;
private static readonly SearchValues<char> ForbiddenChars = SearchValues.Create("<>:\"/\\|?*");
public static AccessLevel LockdownLevel { get; set; }
public static Dictionary<IPAddress, int> IPTable
{
get
{
if (m_IPTable == null)
{
m_IPTable = new Dictionary<IPAddress, int>();
foreach (Account a in Accounts.GetAccounts())
{
if (a.LoginIPs.Length > 0)
{
var ip = a.LoginIPs[0];
m_IPTable[ip] = (m_IPTable.TryGetValue(ip, out var value) ? value : 0) + 1;
}
}
}
return m_IPTable;
}
}
public static void Configure()
{
MaxAccountsPerIP = ServerConfiguration.GetOrUpdateSetting("accountHandler.maxAccountsPerIP", 1);
AutoAccountCreation = ServerConfiguration.GetOrUpdateSetting("accountHandler.enableAutoAccountCreation", true);
PasswordCommandEnabled = ServerConfiguration.GetOrUpdateSetting(
"accountHandler.enablePlayerPasswordCommand",
false
);
if (PasswordCommandEnabled)
{
CommandSystem.Register("Password", AccessLevel.Player, Password_OnCommand);
}
}
public static void Initialize()
{
EventSink.AccountLogin += EventSink_AccountLogin;
EventSink.Shutdown += PasswordWorker.Stop;
EventSink.ServerCrashed += PasswordWorker.OnCrashed;
}
[Usage("Password <newPassword> <repeatPassword>")]
[Description(
"Changes the password of the commanding players account. Requires the same C-class IP address as the account's creator."
)]
public static void Password_OnCommand(CommandEventArgs e)
{
var from = e.Mobile;
if (from.Account is not Account acct)
{
return;
}
var accessList = acct.LoginIPs;
if (accessList.Length == 0)
{
return;
}
var ns = from.NetState;
if (ns == null)
{
return;
}
if (e.Length == 0)
{
from.SendMessage("You must specify the new password.");
return;
}
if (e.Length == 1)
{
from.SendMessage("To prevent potential typing mistakes, you must type the password twice. Use the format:");
from.SendMessage("Password \"(newPassword)\" \"(repeated)\"");
return;
}
var pass = e.GetString(0);
var pass2 = e.GetString(1);
if (pass != pass2)
{
from.SendMessage("The passwords do not match.");
return;
}
var isSafe = true;
for (var i = 0; isSafe && i < pass.Length; ++i)
{
isSafe = pass[i] >= 0x20 && pass[i] < 0x7F;
}
if (!isSafe)
{
from.SendMessage("That is not a valid password.");
return;
}
try
{
var ipAddress = ns.Address;
if (accessList[0].MatchClassC(ipAddress))
{
// Confirmed from the callback: off-loop the write has not landed yet here.
PasswordWorker.SetPassword(
acct,
pass,
_ => from.SendMessage("The password to your account has changed.")
);
}
else
{
var entry = PageQueue.GetEntry(from);
if (entry != null)
{
if (entry.Message.StartsWithOrdinal("[Automated: Change Password]"))
{
from.SendMessage("You already have a password change request in the help system queue.");
}
else
{
from.SendMessage("Your IP address does not match that which created this account.");
}
}
else if (PageQueue.CheckAllowedToPage(from))
{
from.SendMessage(
"Your IP address does not match that which created this account. A page has been entered into the help system on your behalf."
);
/* The next available Counselor/Game Master will respond as soon as possible.
* Please check your Journal for messages every few minutes.
*/
from.SendLocalizedMessage(501234, "", 0x35);
PageQueue.Enqueue(
new PageEntry(
from,
$"[Automated: Change Password]<br>Desired password: {pass}<br>Current IP address: {ipAddress}<br>Account IP address: {accessList[0]}",
PageType.Account
)
);
}
}
}
catch
{
// ignored
}
}
public static void DeleteRequest(NetState state, int index)
{
if (state.Account is not Account acct)
{
state.Disconnect("Attempted to delete a character but the account could not be found.");
return;
}
DeleteResultType res;
if (index < 0 || index >= acct.Length)
{
res = DeleteResultType.BadRequest;
}
else
{
var m = acct[index];
if (m == null)
{
res = DeleteResultType.CharNotExist;
}
else if (m.NetState != null)
{
res = DeleteResultType.CharBeingPlayed;
}
else if (acct.AccessLevel == AccessLevel.Player && RestrictDeletion && Core.Now < m.Created + DeleteDelay)
{
res = DeleteResultType.CharTooYoung;
}
// Don't need to check current location, if netstate is null, they're logged out
else if (
m.AccessLevel == AccessLevel.Player &&
Region.Find(m.LogoutLocation, m.LogoutMap).IsPartOf<JailRegion>()
)
{
res = DeleteResultType.BadRequest;
}
else
{
state.LogInfo($"Deleting character {index} ({m.Serial})");
acct.Comments.Add(new AccountComment("System", $"Character #{index + 1} {m} deleted by {state}"));
m.Delete();
state.SendCharacterListUpdate(acct);
return;
}
}
state.SendCharacterDeleteResult(res);
state.SendCharacterListUpdate(acct);
}
public static bool CanCreate(IPAddress ip) =>
!IPTable.TryGetValue(ip, out var result) || result < MaxAccountsPerIP;
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static bool IsValidUsername(ReadOnlySpan<char> username) =>
username.Length > 0 &&
// Usernames must not start with a space, end with a space, or end with a period
!username.StartsWith(' ') && !username.EndsWith(' ') && !username.EndsWith('.') &&
!username.ContainsAny(ForbiddenChars);
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static bool IsValidPassword(ReadOnlySpan<char> password) => password.Length > 0;
private static Account CreateAccount(NetState state, string username, string password)
{
if (!IsValidUsername(username) || !IsValidPassword(password))
{
return null;
}
if (!CanCreate(state.Address))
{
logger.Information(
$"Login: {{NetState}} Account '{{Username}}' not created, ip already has {{AccountCount}} account{(MaxAccountsPerIP == 1 ? "" : "s")}.",
state,
username,
MaxAccountsPerIP
);
return null;
}
logger.Information("Login: {NetState}: Creating new account '{Username}'", state, username);
return new Account(username, password);
}
public static void EventSink_AccountLogin(AccountLoginEventArgs e)
{
var un = e.Username;
var pw = e.Password;
e.Accepted = false;
if (Accounts.GetAccount(un) is not Account acct)
{
// To prevent someone from making an account of just '' or a bunch of meaningless spaces
if (AutoAccountCreation && !string.IsNullOrWhiteSpace(un))
{
e.State.Account = acct = CreateAccount(e.State, un, pw);
e.Accepted = acct?.CheckAccess(e.State) ?? false;
if (!e.Accepted)
{
e.RejectReason = ALRReason.BadComm;
}
}
else
{
logger.Information("Login: {NetState} Invalid username '{Username}'", e.State, un);
e.RejectReason = ALRReason.Invalid;
}
}
else if (!acct.HasAccess(e.State))
{
logger.Information("Login: {NetState} Access denied for '{Username}'", e.State, un);
e.RejectReason = LockdownLevel > AccessLevel.Player ? ALRReason.BadComm : ALRReason.BadPass;
}
else
{
HandlePasswordCheck(e, acct, pw);
}
}
/// <summary>
/// Separate from the caller's else-if chain because two outcomes are not verdicts: the off-loop
/// path has none yet, and a full queue must reject rather than fall through and verify.
/// </summary>
private static void HandlePasswordCheck(AccountLoginEventArgs e, Account acct, string pw)
{
switch (DispatchPasswordCheck(e, acct, pw))
{
case PasswordCheckDispatch.Deferred:
{
e.Deferred = true;
return;
}
case PasswordCheckDispatch.Saturated:
{
// Reject rather than verify inline: steering work back onto the loop is what a
// flood wants.
logger.Warning(
"Login: {NetState} Password verification queue full, rejecting '{Username}'",
e.State,
acct.Username
);
e.RejectReason = ALRReason.BadComm;
return;
}
}
if (!acct.CheckPassword(pw))
{
logger.Information("Login: {NetState} Invalid password for '{Username}'", e.State, acct.Username);
e.RejectReason = ALRReason.BadPass;
return;
}
ApplyVerifiedLogin(e, acct);
}
/// <summary>Everything after the password is known good, shared so an off-loop verdict lands
/// in the same state as an inline one.</summary>
private static void ApplyVerifiedLogin(AccountLoginEventArgs e, Account acct)
{
if (acct.Banned)
{
logger.Information("Login: {NetState} Banned account '{Username}'", e.State, acct.Username);
e.RejectReason = ALRReason.Blocked;
return;
}
logger.Information("Login: {NetState} Valid credentials for '{Username}'", e.State, acct.Username);
e.State.Account = acct;
e.Accepted = true;
acct.LogAccess(e.State);
LoginAllowlist.RecordLogin(e.State?.Address);
}
private enum PasswordCheckDispatch
{
/// <summary>Verify on the loop.</summary>
Inline,
/// <summary>Handed to the worker; no verdict yet.</summary>
Deferred,
/// <summary>The queue is full.</summary>
Saturated
}
/// <summary>
/// Hands the password check to the worker, whatever algorithm it uses. Every protection is safe
/// off the loop, so there is no carve-out, and a cheap digest does not need one either:
/// <c>AccountSecurity.Configure</c> refuses anything below SHA2 as the configured algorithm, so
/// MD5 and SHA1 only appear as a stored hash awaiting migration. That makes
/// <c>NeedsPasswordUpgrade</c> true, and the upgrade hash dominates the job.
/// </summary>
private static PasswordCheckDispatch DispatchPasswordCheck(AccountLoginEventArgs e, Account acct, string pw)
{
if (!PasswordWorker.Enabled)
{
return PasswordCheckDispatch.Inline;
}
var job = new PasswordJob
{
Account = acct,
State = e.State,
StoredHash = acct.Password,
StoredAlgorithm = acct.PasswordAlgorithm,
VerifyPhrase = acct.GetVerifyPhrase(pw),
HashPhrase = acct.NeedsPasswordUpgrade() ? acct.GetRehashPhrase(pw) : null,
TargetAlgorithm = AccountSecurity.CurrentAlgorithm,
OnComplete = static (j, outcome) =>
CompleteDeferredAccountLogin(j.State, j.Account, outcome.Verified)
};
return PasswordWorker.TryEnqueue(job)
? PasswordCheckDispatch.Deferred
: PasswordCheckDispatch.Saturated;
}
/// <summary>Resumes a login whose password check ran on the verification thread.</summary>
internal static void CompleteDeferredAccountLogin(NetState state, Account acct, bool verified)
{
var e = new AccountLoginEventArgs(state, acct.Username, null);
if (verified)
{
ApplyVerifiedLogin(e, acct);
}
else
{
logger.Information("Login: {NetState} Invalid password for '{Username}'", state, acct.Username);
e.RejectReason = ALRReason.BadPass;
}
IncomingAccountPackets.CompleteAccountLogin(state, e.Accepted, e.RejectReason);
}
[OnEvent(nameof(GameServer.GameServerLoginEvent))]
public static void OnGameServerLogin(GameServer.GameLoginEventArgs e)
{
var un = e.Username;
var pw = e.Password;
if (Accounts.GetAccount(un) is not Account acct)
{
e.Accepted = false;
}
else if (!acct.HasAccess(e.State))
{
logger.Information("Login: {NetState} Access denied for '{Username}'", e.State, un);
e.Accepted = false;
}
// The auth id was only issued after the account login packet verified this password, so
// re-deriving the hash costs a second Argon2 verify to answer the same question.
else if (!e.PreAuthenticated && !acct.CheckPassword(pw))
{
logger.Information("Login: {NetState} Invalid password for '{Username}'", e.State, un);
e.Accepted = false;
}
else if (acct.Banned)
{
logger.Information("Login: {NetState} Banned account '{Username}'", e.State, un);
e.Accepted = false;
}
else
{
acct.LogAccess(e.State);
LoginAllowlist.RecordLogin(e.State?.Address);
logger.Information("Login: {NetState} Account '{Username}' at character list", e.State, un);
e.State.Account = acct;
e.Accepted = true;
e.CityInfo = CharacterCreation.GetStartingCities();
}
}
public static bool CheckAccount(Mobile mobCheck, Mobile accCheck)
{
if (accCheck?.Account is Account a)
{
for (var i = 0; i < a.Length; ++i)
{
if (a[i] == mobCheck)
{
return true;
}
}
}
return false;
}
}