ModernUO/Projects/UOContent/Accounting/Security/PasswordWorker.cs
Kamron Batman a7e65aab01
perf(login): run password hashing on a parked worker thread (#2566)
## Why

An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.

## What the measurement says

Off-loading does not delete the cost, it relocates it. Three things stay on the loop:

| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |

Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.

Two results shaped the design:

- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.

## Why exactly one worker

It is load-bearing three times over, which is also why it must not quietly become a pool:

- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.

Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.

## Making every protection safe off-thread

The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:

| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |

Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.

With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".

## Correctness

- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.

## Bounding

`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.

The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.

## Gate

Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.

## Engine change

One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.

## Docs

`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.

## Testing

698 UOContent tests, 810 Server tests, Release build clean.

Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
2026-08-09 00:13:34 -07:00

293 lines
11 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: PasswordWorker.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
using System;
using System.Collections.Concurrent;
using System.Threading;
using Server.Logging;
using Server.Network;
namespace Server.Accounting.Security;
/// <summary>
/// Work handed to the password thread, which reads no game state and writes none.
///
/// Verify and hash are independently optional: a login verifies and may rehash, an explicit change
/// only hashes.
/// </summary>
internal sealed class PasswordJob
{
public Account Account;
/// <summary>Ties the job to a connection. Null when the work is not gated on one, such as a
/// password change by an admin.</summary>
public NetState State;
/// <summary>Hash to verify against, with <see cref="VerifyPhrase"/>.</summary>
public string StoredHash;
/// <summary>Algorithm <see cref="StoredHash"/> was written with. Both algorithms are resolved on
/// the loop; <c>AccountSecurity.CurrentAlgorithm</c> is mutable state the worker must not read.</summary>
public PasswordProtectionAlgorithm StoredAlgorithm;
/// <summary>Phrase to verify, or null to skip verification.</summary>
public string VerifyPhrase;
/// <summary>Phrase to hash, or null when nothing needs writing.</summary>
public string HashPhrase;
public PasswordProtectionAlgorithm TargetAlgorithm;
/// <summary>Runs on the game loop with the result. Free to touch game state.</summary>
public Action<PasswordJob, PasswordOutcome> OnComplete;
}
internal readonly struct PasswordOutcome
{
/// <summary>True when no verification was asked for, or it succeeded.</summary>
public readonly bool Verified;
/// <summary>The derived hash, or null when nothing was hashed or verification failed.</summary>
public readonly string Hash;
public PasswordOutcome(bool verified, string hash)
{
Verified = verified;
Hash = hash;
}
}
/// <summary>
/// Runs password hashing off the game loop. An Argon2 verify costs ~8.9 ms of frozen world per
/// login attempt, successful or not.
///
/// Exactly one worker, and that is load-bearing three times over. It cannot cost the loop more than
/// an inline verify under any scheduling regime, because at worst it takes an equal share of one
/// core -- which is what lets the measurement hold on hardware we cannot inspect. It caps live
/// hashing arenas at one. And writes apply in dispatch order only because a single thread drains
/// FIFO, so a second would need ordering reintroduced.
///
/// ~110 verifies/sec, which is ample: only loop time matters, not login latency.
/// </summary>
internal sealed class PasswordWorker
{
private static readonly ILogger logger = LogFactory.GetLogger(typeof(PasswordWorker));
/// <summary>
/// Backstop, not a flood defense. <c>SentFirstPacket</c> holds a connection to one pending
/// verify and the engine caps connections at 4096 (<c>NetState.Network.cs</c>), so this matches
/// that bound and can only trip if that invariant breaks. A cap low enough to blunt an attack
/// would reject real players first; flood defense belongs at the connection layer.
/// </summary>
private const int MaxPending = 4096;
// Nothing signals the worker when a save freeze ends, so it re-checks on this interval -- but
// only while a save is in progress, never in steady state.
private const int SaveGatePollMs = 50;
private static PasswordWorker _instance;
// Needs a spare core to move work to, which a 1-2 core host does not have. Off in DEBUG, where
// logins are rare and the inline path is easier to follow.
internal static readonly bool Enabled =
#if DEBUG
false;
#else
Environment.ProcessorCount >= 4;
#endif
private readonly Thread _thread;
private readonly AutoResetEvent _work = new(false);
private readonly ConcurrentQueue<PasswordJob> _queue = [];
private int _pending;
private volatile bool _exit;
private PasswordWorker()
{
_thread = new Thread(Execute)
{
IsBackground = true,
Name = "Password Worker"
};
_thread.Start();
}
private static PasswordWorker Instance => _instance ??= new PasswordWorker();
/// <summary>Queues a job. False when full, and the caller must then reject without verifying.</summary>
internal static bool TryEnqueue(PasswordJob job) => Instance.TryEnqueueCore(job);
private bool TryEnqueueCore(PasswordJob job)
{
if (Volatile.Read(ref _pending) >= MaxPending)
{
return false;
}
Interlocked.Increment(ref _pending);
_queue.Enqueue(job);
_work.Set();
return true;
}
/// <summary>
/// Checked before each job, which bounds a save overlap to whichever hash was already running:
/// the freeze holds the loop, so nothing new can be queued during it. PendingSave counts too --
/// the serialization threads are already awake and spinning on an empty queue by then.
/// </summary>
private static bool CanRunNow() => World.WorldState is WorldState.Running or WorldState.WritingSave;
private void Execute()
{
while (!_exit)
{
if (_queue.IsEmpty)
{
// A kernel block at zero CPU. Set() during a hash leaves the event signalled, so a
// wake arriving mid-job is not lost.
_work.WaitOne();
continue;
}
if (!CanRunNow())
{
_work.WaitOne(SaveGatePollMs);
continue;
}
if (!_queue.TryDequeue(out var job))
{
continue;
}
Interlocked.Decrement(ref _pending);
// Gone while it waited: skip it rather than hash for a verdict nobody receives. Running
// only goes true -> false, so a stale read wastes a hash but never skips a live one. A
// null State is a job with no connection to lose, and still runs.
if (job.State?.Running == false)
{
continue;
}
PasswordOutcome outcome;
try
{
outcome = Compute(job);
}
catch (Exception ex)
{
// A verdict must still come back, or the connection never gets a reply.
logger.Error(ex, "Password work failed for {Username}", job.Account?.Username);
outcome = new PasswordOutcome(false, null);
}
Core.LoopContext.Post(() => Apply(job, outcome));
}
}
private static PasswordOutcome Compute(PasswordJob job)
{
if (job.VerifyPhrase != null &&
!AccountSecurity.GetPasswordProtection(job.StoredAlgorithm)
.ValidatePassword(job.StoredHash, job.VerifyPhrase))
{
return new PasswordOutcome(false, null);
}
return new PasswordOutcome(
true,
job.HashPhrase == null
? null : AccountSecurity.GetPasswordProtection(job.TargetAlgorithm).EncryptPassword(job.HashPhrase)
);
}
private static void Apply(PasswordJob job, PasswordOutcome outcome)
{
// Re-checked: a connection can drop while the result sits in the loop queue.
if (job.State?.Running == false)
{
return;
}
if (outcome.Verified && outcome.Hash != null)
{
job.Account.ApplyPasswordWrite(outcome.Hash, job.TargetAlgorithm);
}
job.OnComplete?.Invoke(job, outcome);
}
/// <summary>
/// Sets a password, off the loop where available and inline otherwise, invoking
/// <paramref name="onDone"/> on the loop either way.
///
/// Confirm from <paramref name="onDone"/>, not the call site: off-loop the write has not
/// happened when this returns.
/// </summary>
internal static void SetPassword(Account account, string plainPassword, Action<bool> onDone)
{
if (!Enabled)
{
account.SetPassword(plainPassword);
onDone?.Invoke(true);
return;
}
var job = new PasswordJob
{
Account = account,
HashPhrase = account.GetRehashPhrase(plainPassword),
TargetAlgorithm = AccountSecurity.CurrentAlgorithm,
OnComplete = (_, outcome) => onDone?.Invoke(outcome.Hash != null)
};
if (!TryEnqueue(job))
{
// Saturated. Unlike a login, a password change must not be dropped, so it pays the
// hash on the loop instead.
account.SetPassword(plainPassword);
onDone?.Invoke(true);
}
}
/// <summary>Runs a job on the calling thread. The seam the tests drive.</summary>
internal static PasswordOutcome ComputeInline(PasswordJob job) => Compute(job);
/// <summary>
/// Stops the worker on shutdown or crash. Pending jobs are dropped rather than finished:
/// nothing saves the world after this point, so a write applied here would reach no disk.
///
/// Draining the loop context is not this type's business either. That belongs in the core
/// shutdown path, before subscriber events run -- a subscriber pumping the shared context would
/// execute other subscribers' work at an arbitrary point in the event order.
/// </summary>
internal static void Stop() => _instance?.StopThread();
// HandleClosed skips InvokeShutdown when the server crashed, so the crash path needs its own
// subscription.
internal static void OnCrashed(ServerCrashedEventArgs e) => Stop();
private void StopThread()
{
_exit = true;
_work.Set();
_thread.Join(TimeSpan.FromSeconds(5));
}
}