ModernUO/dev-docs/claude-skills/modernuo-threading.md
Kamron Batman a7e65aab01
perf(login): run password hashing on a parked worker thread (#2566)
## Why

An Argon2 verify is **~8.9 ms of frozen world per login attempt** — more than half a 16 ms frame. Failed attempts cost exactly the same as successful ones, by design, so a credential-stuffing flood is a full-cost stall per packet without needing valid credentials. `SetPassword` derives a hash too, so `[password`, the admin gump and account creation each pay the same.

## What the measurement says

Off-loading does not delete the cost, it relocates it. Three things stay on the loop:

| Component | Measured |
|---|---:|
| Inline verify (today) | **8.92 ms** |
| Dispatch to the worker | 210 ns |
| Drain the continuation off `LoopContext` | 13 ns |
| Loop's own work slowed by shared-L3 eviction | **0.05 – 5.44 ms** |

Net gain **3.5 – 8.9 ms** of on-loop time per login. Harness in `ModernUO-Benchmarks` (`Benchmarks/Argon2OffLoop/`): it models the loop as a dependent-load pointer chase swept across working-set sizes, which is an upper bound on cache-latency sensitivity, and copies `EventLoopContext` so the hand-off cost is the real one.

Two results shaped the design:

- **The contention tax peaks in the middle of the working-set range**, not at the top — 5.44 ms at 8 MiB (a quarter of this chip's L3), but 0.76 ms at 30 MiB and 0.10 ms at 256 KiB. A tiny hot set has nothing in L3 to lose; a huge one is already DRAM-bound.
- **Per-login tax falls as concurrency rises** (5.44 → 2.56 → 1.60 ms at 1/2/4 hashers) while *total* loop damage rises. Contention is shared, not additive, so a login rush is not the disaster case — a single login is.

## Why exactly one worker

It is load-bearing three times over, which is also why it must not quietly become a pool:

- **Cost bound.** Off-loop loses to inline only if a hash steals ~82% of the loop's throughput. One hasher contending for one core leaves the loop ~50%. **A single background hasher cannot cost the loop more than the inline verify under any scheduling regime**, which is what lets the measurement hold on hardware we cannot inspect — AMD, VPS, oversubscribed VM. Four hashers drop the loop to ~20% and break it.
- **Memory.** Exactly one hashing arena is live at a time whatever the login volume.
- **Ordering.** Writes apply in dispatch order *only* because a single thread drains FIFO. A second worker would need ordering reintroduced; `WritesApplyInDispatchOrder` fails if that happens.

Throughput is ~110 verifies/sec. Only loop time matters, not login latency, so head-of-line blocking during a rush costs nothing.

## Making every protection safe off-thread

The worker was initially Argon2-only. That was the right call for the wrong reason — it was blamed on Argon2's salt RNG, which is a stateless syscall wrapper and was never a problem. The real blockers were elsewhere, and both are fixed at the source:

| Protection | Was | Now |
|---|---|---|
| MD5/SHA1/SHA2 | shared `HashAlgorithm.ComputeHash`, which carries the running digest across `HashCore`/`HashFinal` through process-wide singletons | static `HashData` into a `stackalloc` span — no state, no allocation, identical bytes |
| PBKDF2 | `Utility.RandomMinMax` → shared `System.Random`, thread-unsafe *and* game state | `RandomNumberGenerator.GetInt32`, matching the salt beside it |
| Argon2 | already safe (`Verify` is static + stackalloc) | unchanged, singleton reused |

Literal digests are pinned in a test **before** the change and still pass after it. These are compared as strings against every account database, so any casing or encoding drift would lock out every SHA and MD5 account at once.

With all three safe, the worker no longer knows which algorithm it runs and the dispatch conditions collapse to "is off-loop available".

## Correctness

- **Phrase derivation** moves to `AccountSecurity.DerivePhrase`, so verification (stored algorithm's rule) and rehash (target algorithm's rule) cannot disagree. Deriving with the wrong one is the shape of the lockout fixed in #2562.
- **Liveness** is checked at dequeue *and* at apply — a connection can drop while queued or while the result sits in the loop queue. A job with no connection attached, such as an admin password change, runs regardless.
- **Queue overflow rejects** a login rather than verifying inline; steering work back onto the loop is what a flood wants. A password change instead falls back to hashing inline, because unlike a login it must not be dropped.
- **Shutdown and crash** both just stop the thread, and pending jobs are dropped. No save is initiated once shutdown begins — saving is the operator's choice up front, via the admin gump's save/no-save variants, and `WaitForWriteCompletion` honours one already in flight — so a write applied during teardown would reach no disk. The crash path needs its own subscription because `HandleClosed` skips `InvokeShutdown` when crashed.

## Bounding

`MaxPending` is 4096 — a backstop, not a flood defense. `SentFirstPacket` holds a connection to one pending verify and the engine caps connections at 4096, so the queue is already bounded by construction and this can only trip if that invariant breaks. A cap low enough to blunt an attack would reject real players first; during a mass reconnect they *are* the queue. Flood defense belongs at the connection layer.

The real DoS improvement is elsewhere: today every attempt stalls the world, and after this a flood occupies one core while the loop keeps ticking.

## Gate

Release builds on 4+ cores. Below that there is no spare core to move work to, so off-loading buys nothing by construction; `DEBUG` is excluded because dev boxes and test shards have few logins. Both modes call the same code — the gate only chooses where it runs.

## Engine change

One property, `AccountLoginEventArgs.Deferred`, so a subscriber can say "no verdict yet". `EventSink.AccountLogin` is `Action<...>` with no continuation, and the packet handler replies in the same call. Approved separately since it touches `Projects/Server/`.

## Docs

`dev-docs/threading-model.md` and the threading skill gain a vetted-workers section. The forbidden-patterns table bans `new Thread`, `ConcurrentQueue<T>`, `Interlocked` and `volatile` in `UOContent`, and its exceptions covered only `Projects/Server/` — the existing Advanced Search fan-out already sat outside it. The new section leads with proving the need (measure on-loop time, not wall-clock; gate on core count; record the measurement), keeps game logic on the loop via chunking, and documents the hand-off protocol in both directions.

## Testing

698 UOContent tests, 810 Server tests, Release build clean.

Covered: verify and rehash outcomes, phrase rules for SHA1/SHA2 vs Argon2, stored-format stability for MD5/SHA1/SHA2, jobs with no connection attached, and dispatch ordering through the real queue. The liveness and ordering guards are mutation-verified.
2026-08-09 00:13:34 -07:00

9.8 KiB

name description
modernuo-threading Trigger when discussing async patterns, world saves, game loop, or reviewing code for threading issues. When using await, Task, or any concurrency-related code in game logic.

ModernUO Threading & Event Loop

When This Activates

  • Reviewing code for threading issues
  • Discussing async/await patterns
  • Working with world saves
  • Any mention of Task.Run, Thread, lock, ConcurrentDictionary
  • Understanding the game loop

CRITICAL RULE: Single-Threaded Game Logic

ModernUO uses a single-threaded game loop. All game logic runs on one thread. There are NO exceptions for game code.

Forbidden in Game Code

// ALL of these are WRONG in Projects/UOContent/ code:
Task.Run(() => ProcessItems());           // Background thread
new Thread(BackgroundWork).Start();       // Manual thread
ThreadPool.QueueUserWorkItem(Work);       // Thread pool
lock (_syncObj) { ... }                   // Locking
Monitor.Enter(obj);                       // Monitor
volatile int _counter;                    // Volatile
ConcurrentDictionary<int, Item> _items;   // Concurrent collections
ConcurrentQueue<T> _queue;               // Concurrent collections
Interlocked.Increment(ref _count);        // Atomics
Mutex mutex;                              // OS mutex
Semaphore sem;                            // Semaphore
ReaderWriterLockSlim rwl;                 // RW lock

Why: The game loop is single-threaded. Concurrency primitives add overhead for no benefit, and background threads would cause data races with game state.

Why await Is Safe

EventLoopContext implements SynchronizationContext and routes all await continuations back to the main thread:

// This is SAFE in game code:
await Timer.Pause(TimeSpan.FromMilliseconds(100));
// Continuation runs on the game thread, not a thread pool thread

The flow:

  1. await captures EventLoopContext as the synchronization context
  2. When the awaited task completes, the continuation is posted to EventLoopContext._queue
  3. LoopContext.ExecuteTasks() runs those continuations on the main thread during the next game loop tick

Game Loop Structure

// Simplified from Projects/Server/Main.cs
while (!Closing)
{
    _tickCount = GetTimestamp();
    _now = DateTime.UtcNow;

    Mobile.ProcessDeltaQueue();    // Send mobile state changes to clients
    Item.ProcessDeltaQueue();      // Send item state changes to clients
    Timer.Slice(_tickCount);       // Execute due timers
    NetState.Slice();              // Process network I/O
    LoopContext.ExecuteTasks();     // Run async continuations (Timer.Pause, etc.)
    Timer.CheckTimerPool();        // Refill timer pool if needed
}

EventLoopContext Details

public sealed class EventLoopContext : SynchronizationContext
{
    private readonly ConcurrentQueue<Action> _queue;          // Normal tasks
    private readonly ConcurrentQueue<Action> _priorityQueue;  // High priority
    private readonly int _maxPerFrame;                         // Default: 128

    // Posts run on next ExecuteTasks() call
    public void Post(Action d, Priority priority = Priority.Normal);

    // Send blocks if called from another thread, immediate if on game thread
    public override void Send(SendOrPostCallback d, object state);

    // Called once per game loop tick
    public void ExecuteTasks();
}

Memory: STArrayPool vs ArrayPool

In game code, use STArrayPool<T>.Shared (single-threaded, no locks):

// GOOD - no locking overhead
var buffer = STArrayPool<byte>.Shared.Rent(1024);
try { /* use buffer */ }
finally { STArrayPool<byte>.Shared.Return(buffer); }

ArrayPool<T>.Shared uses locks for thread safety -- unnecessary overhead in single-threaded context.

Memory: PooledRefList

// Stack-allocated list using pooled arrays
using var list = PooledRefList<Mobile>.Create();
list.Add(mobile);
// Automatically returns array to pool on Dispose

// For multi-threaded contexts (rare):
using var list = PooledRefList<Mobile>.CreateMT();

World Saves

World saves use parallel serialization threads. This is critical to understand:

  1. Preserialize: Allocates heaps and wakes serialization thread workers (background)
  2. Snapshot: Main thread calls Persistence.SerializeAll() which pushes entities into SerializationThreadWorker queues (round-robin). Workers call Serialize(writer) on their own background threads in parallel.
  3. Write snapshot: Disk I/O on background threads after serialization completes
// From World.cs -- save flow:
World.Save();
 Preserialize() on thread pool (allocate heaps, wake serialization workers)
 Snapshot() on main thread (queues entities to workers, workers serialize in parallel)
   SerializationThreadWorker.Execute() calls e.Serialize(writer) on background thread
 PauseSerializationThreads() (wait for workers to finish)
 WriteSnapshot() on thread pool (disk I/O only)

Serialize() runs on background threads

Because SerializationThreadWorker calls Serialize() on its own thread, Serialize() must be pure:

  • NO creating/destroying Items or Mobiles
  • NO starting/stopping timers (not thread-safe)
  • NO sending packets or modifying NetState
  • NO mutating shared game state
  • ONLY read fields and write to IGenericWriter

See modernuo-serialization.md for full purity rules.

Exceptions: Server Infrastructure

These files MAY use threading (they're server infrastructure, not game logic):

  • Projects/Server/Main.cs - Event loop, thread setup
  • Projects/Server/World/World.cs - World save I/O
  • Projects/Server/Network/ - Network I/O
  • Projects/Server/Timer/Timer.Pool.cs - Pool refill

Exceptions: Vetted Workers in UOContent

A background thread is a last resort. The forbidden list is about game logic, which is never threaded. A dedicated worker touching no game state is the sanctioned way off the loop, and necessarily uses new Thread, ConcurrentQueue<T>, Interlocked, AutoResetEvent and volatile at the thread boundary only.

Prove the need first

  • Measure on-loop time, not wall-clock. Frozen world is the cost; player latency is not.
  • Off-loading creates no CPU. On 1-2 cores there is no spare core — gate on ProcessorCount.
  • Count what stays: dispatch, continuation, and the loop slowing while the worker evicts shared L3.
  • Record the measurement, or nobody can re-justify the worker later.

Game logic stays on the loop — chunk it

Work needing game state cannot be threaded at any core count. Too slow for one tick? Split across ticks, bounded by count or elapsed time — never "until done".

Timer.DelayCall(TimeSpan.Zero, TimeSpan.FromMilliseconds(50), () =>
{
    var budget = 0;
    while (_cursor < _items.Count && budget++ < 100) { Process(_items[_cursor++]); }
});

Vetted workers

Worker Justification
Accounting/Security/PasswordWorker.cs 8.9 ms/login on-loop at Argon2; 3.5-8.9 ms measured saving
Engines/Advanced Search/AdvancedSearchGump.cs Admin-triggered full-world scan, saves disabled

The six rules

  1. No game state read or written off-thread; dispatch immutable values captured on the loop.
  2. Resolve policy (algorithm, salt, era branch) at dispatch — the worker holds none.
  3. Park on a kernel wait, never spin. Spinning burns a core on shared hosts.
  4. Run only while WorldState is Running or WritingSave. Not World.Saving — that misses PendingSave, where serialization threads are already spinning.
  5. Bounded queue, or a bound upstream named in a comment.
  6. Everything the worker calls must itself be thread-safe. A singleton is not automatically safe — HashAlgorithm.ComputeHash carries state, Utility's RNG is a shared System.Random and game state. Prefer static one-shot APIs (SHA256.HashData, RandomNumberGenerator.Fill).

Crossing the boundary

Dispatch captures what the continuation will need to re-validate:

var job = new Job { Target = state, Expected = account.Password, Input = DerivePhrase(...) };
if (!Worker.TryEnqueue(job)) { /* reject — never fall back to running it inline */ }

Hand back one of two ways, and no other:

Core.LoopContext.Post(() => Apply(job, result));  // a result for a specific caller
Volatile.Write(ref _snapshot, newTable);          // a shared table rebuilt periodically

The continuation re-validates, because time passed:

if (job.Target?.Running != true) { return; }                     // gone
if (account.Password != job.Expected) { return; }                // changed underneath

Always post a result, including on failure — a worker that throws silently leaves its caller waiting forever. Use ConfigureAwait(false) on every await inside off-loop work.

Anti-Patterns

Pattern Problem Solution
Task.Run(...) Runs on thread pool, races with game state Use Timer.StartTimer()
new Thread(...) Same as above Use Timer.StartTimer()
lock(obj) Unnecessary overhead, no contention exists Remove lock, use plain code
ConcurrentDictionary Lock-free but still overhead Use Dictionary<K,V>
volatile Memory barriers not needed on single thread Use plain field
Thread.Sleep() Blocks entire game loop Use await Timer.Pause()
ArrayPool<T>.Shared Uses locks Use STArrayPool<T>.Shared

Real Examples

  • Game loop: Projects/Server/Main.cs (RunEventLoop)
  • EventLoopContext: Projects/Server/EventLoopTasks.cs
  • STArrayPool: Projects/Server/Buffers/STArrayPool.cs
  • PooledRefList: Projects/Server/Collections/PooledRefList.cs
  • World save: Projects/Server/World/World.cs

See Also

  • dev-docs/threading-model.md - Complete threading documentation
  • dev-docs/claude-skills/modernuo-code-audit.md - Threading audit rules
  • dev-docs/claude-skills/modernuo-timers.md - Timer-based scheduling