ModernUO/dev-docs/claude-skills/modernuo-code-audit.md
Kamron Batman 6d846b11e5
perf: Sleep the event loop when idle. Fixes networking micro-stalls. Adds event loop instrumentation. (#2559)
## Problem

`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.

## Result

Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:

| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |

**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.

## How

The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.

**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.

**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.

## Configuration

| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |

## Diagnostics (compiled out by default)

`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.

## Verification

- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.

## Notes

The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
2026-08-09 13:24:59 -07:00

15 KiB

name description
modernuo-code-audit Auto-trigger whenever writing or modifying .cs files under Projects/. Audits code for ModernUO convention violations. Warnings only - flag issues and ask before fixing.

ModernUO Code Audit

When This Activates

  • Any time you write, edit, or modify a .cs file under Projects/
  • After generating code snippets for the user
  • During code review

Audit Rules (Warnings Only)

Flag these issues but do NOT auto-fix. Ask the user before making changes.

1. LINQ: Know What's Optimized (.NET 10)

Not all LINQ is banned. .NET 10 JIT/PGO eliminates overhead for specific patterns. Anything not listed below is still forbidden on hot paths.

Tier 1 — Zero-cost (use freely on hot paths):

  • foreach over IEnumerable<T> backed by T[], List<T>, Stack<T>, Queue<T> — PGO devirtualizes the enumerator, zero heap allocation
  • .Contains() after a preceding LINQ operator (.Distinct(), .OrderBy(), .Reverse(), .Union(), .Intersect(), .Except(), .Concat(), .SelectMany(), .Where().Select(), .Skip(), .Take(), .OfType(), .Cast(), .Shuffle()) — LINQ has ~30 specialized overrides that skip the intermediate work (no sort, no HashSet, no buffering)
  • .Count() on sized collections (ICollection<T>, or after Range/Repeat/Skip/Take/Append) — O(1) property access, no enumeration
  • .OrderBy().First() / .OrderByDescending().First() / .OrderBy().Last() — O(N) min/max scan, no sort performed
  • .Shuffle().Take(n) — reservoir sampling, single pass, O(n) memory
  • Enumerable.Range() / Enumerable.Sequence() followed by .Count(), .Contains(), .ToArray(), .ToList(), .ElementAt(), .Last() — arithmetic, not enumeration

Tier 2 — Low overhead (acceptable on warm paths, benchmark if critical):

  • .Skip(n).Take(m).ToArray() on T[]/List<T> — vectorized Span<T>.CopyTo (still allocates output)
  • .LeftJoin() / .RightJoin() — ~2x faster than manual GroupJoin+SelectMany+DefaultIfEmpty
  • .Where(predicate) on T[]/List<T>WhereIterator still heap-allocates, but enumeration is PGO-optimized. Manual foreach+if is still faster for true hot paths.

Tier 3 — Still forbidden on hot paths (write manual code):

  • .Select(f).Where(p) (this order — each intermediate iterator allocates)
  • .GroupBy(), .ToDictionary(), .ToHashSet(), .ToLookup() (always allocate internal structures)
  • .Aggregate() (delegate overhead per element)
  • .Sum() / .Min() / .Max() on float/double (no SIMD in LINQ on ARM)
  • .SelectMany() when iterating results (not .Contains()) — multiple enumerator allocations
  • .Zip() when iterating — enumerator allocations
  • Any LINQ over IAsyncEnumerable<T> — no PGO/escape analysis
  • Long chains like .Where().Select().OrderBy().Take() — each step allocates an iterator

Prerequisites: .NET 10, tiered compilation + Dynamic PGO enabled (default). Tier 1 optimizations require ~30+ calls for JIT warmup.

Quick decision: If the exact pattern is in Tier 1 → use it. If it's in Tier 2 → acceptable unless profiling shows it's a bottleneck. If it's anything else → manual for/foreach + PooledRefList<T>.

2. No Console.WriteLine

Bad: Console.WriteLine(...), Console.Write(...) Good: private static readonly ILogger logger = LogFactory.GetLogger(typeof(MyClass)); then logger.Information(...), logger.Warning(...), logger.Error(...) Requires: using Server.Logging;

3. No Concurrency Primitives in Game Code

Bad: ConcurrentDictionary, ConcurrentQueue, ConcurrentBag, volatile, lock(...), Mutex, Semaphore, Monitor, Interlocked, ReaderWriterLock Why: Server is single-threaded. These add overhead for no benefit. Instead: Use regular Dictionary<K,V>, List<T>, plain fields.

4. Never Iterate World.Mobiles or World.Items Directly

Bad: foreach (var m in World.Mobiles.Values), World.Items.Values.Where(...) Good: map.GetMobilesInBounds<T>(bounds), map.GetMobilesInRange<T>(point, range), map.GetItemsInRange<T>(point, range) Why: Full world iteration is O(n) over all entities. Spatial queries use sector indexing.

5. Clean Up References in OnDelete/OnAfterDelete

Check: Classes with Item or Mobile references should clean them in OnDelete() or OnAfterDelete(). Pattern:

public override void OnAfterDelete()
{
    _someReference = null;
    base.OnAfterDelete();
}

6. Cancel Timers in OnDelete/OnAfterDelete

Check: Any class with TimerExecutionToken or Timer fields must cancel them on deletion. Pattern:

public override void OnAfterDelete()
{
    _timerToken.Cancel();  // For TimerExecutionToken
    _timer?.Stop();        // For Timer references
    _timer = null;
    base.OnAfterDelete();
}

7. Use STArrayPool, Not ArrayPool

Bad: ArrayPool<T>.Shared.Rent(...) in game logic Good: STArrayPool<T>.Shared.Rent(...) in game logic Why: STArrayPool is single-threaded optimized (no locks). Use ArrayPool only in explicitly multi-threaded code. Also: Always return rented arrays in a finally block.

8. No new List in Hot Paths

Bad: var list = new List<Mobile>(); in frequently-called methods Good: using var list = PooledRefList<Mobile>.Create(); Why: PooledRefList uses pooled arrays, zero GC pressure. It's a ref struct (stack-allocated).

9. Serialization Class Requirements

Check: Classes with [SerializationGenerator] MUST be partial. Check: [Constructible] on parameterless constructors for items/mobiles. Check: TimerExecutionToken fields must NOT have [SerializableField]. Check: Use using ModernUO.Serialization; when using serialization attributes.

10. No Task.Run or new Thread

Bad: Task.Run(...), new Thread(...), ThreadPool.QueueUserWorkItem(...) in game code Why: Game logic runs on the single-threaded event loop. Background threads cause race conditions. Exception: Server infrastructure code (Projects/Server/Main.cs, World saves) may use threading.

11. Never Assume Era

Check: If code uses era-conditional logic (Core.AOS, Core.SE, etc.) and the user hasn't specified a target era, ASK which expansion to target. Why: Different eras have dramatically different mechanics.

12. Naming Conventions

Check: _camelCase for private fields, PascalCase for properties/methods/classes. Note: Legacy code may use m_ prefix -- don't flag existing m_ fields but use _ for new code.

13. No Empty Gumps

Check: Any gump (legacy Gump constructor, or BuildLayout) must not have a code path that produces zero visual elements (no AddBackground, no AddPage with content, etc.). Why: The client has no way to close an empty gump — no close button, no right-click dismiss. This leaks a gump slot on both client and server until relog. Common cause: Early return in a constructor or BuildLayout when prerequisites aren't met. Fix: Use a static DisplayTo(Mobile from) method that validates prerequisites before constructing the gump. Make the constructor private. See Projects/UOContent/Gumps/Go/GoGump.cs for the canonical pattern.

14. PropertyList String Literals Must Be Holes

Check: In any IPropertyList.Add() interpolated string, string constants must be wrapped as holes {"text"}, not bare literals. Bad: list.Add(1060658, $"Chances\t{_charges}"); — "Chances" becomes a delimiter, not an argument. Good: list.Add(1060658, $"{"Chances"}\t{_charges}"); — "Chances" is an argument. Why: The handler treats bare text as delimiters and {} contents as arguments. The property list system is used beyond the game client (e.g., web rendering) which must distinguish arguments from delimiters. Only \t should be a bare literal. Also: If you don't know the text for a cliloc number, see Projects/Server/Localization/Localization.cs LoadClilocs() to learn the binary format, and ask the user where their cliloc.enu file is.

15. Braces Required on All Control Flow

Check: ALL if, else, for, foreach, while, do, switch statements must have braces, even for single-line bodies. Bad:

if (condition)
    DoSomething();

Good:

if (condition)
{
    DoSomething();
}

Why: Reduces merge conflicts and diff sizes.

16. Prefer Switch Expressions and Switch-When Patterns

Check: Where a chain of if/else if maps inputs to outputs, prefer a switch expression. Where pattern matching with guards improves clarity, prefer switch-when. Bad:

if (type == GemType.StarSapphire) return "star sapphire";
else if (type == GemType.Emerald) return "emerald";
else return "gem";

Good:

return type switch
{
    GemType.StarSapphire => "star sapphire",
    GemType.Emerald      => "emerald",
    _                    => "gem"
};

Why: Switch expressions enable JIT/PGO optimization and improve readability. Exception: Skip if the switch would be unreadable or the code is on a cold path.

17. Interpolation Anti-Patterns (handler-aware APIs)

Context: Many ModernUO APIs accept ref RawInterpolatedStringHandler (Mobile.SendMessage/Say/Emote/etc., Item.Public/Local/NonlocalOverheadMessage/SendLocalizedMessageTo/SendMessageTo, IPropertyList.Add, SpanWriter.WriteAscii/WriteLatin1, gump AddLabel/AddHtml/AddHtmlLocalized, Html.Center/Color/Right). The handler overload renders the interpolation directly into a pooled buffer with zero string allocation — but only when the call-site argument is a $"..." literal directly in the parameter slot.

Check: Flag any of the following patterns when the call target is one of those handler-aware APIs. The handler overload is silently bypassed and a string is allocated per call.

Pattern Fix
Send(cond ? $"a" : $"b") if/else with two calls
Send(thing switch { 1 => $"a", _ => $"b" }) switch statement, call per arm
var s = $"foo {x}"; Send(s); (single-use) Inline at call site
Send($"x {value.ToString()}") Drop .ToString() — handler formats directly
Send($"x {td.String()}") Drop .String() — pass td directly
Send($"x {a + b}") (string concat) Multiple holes: Send($"x {a}{b}")
Send(string.Format("x {0}", v)) Send($"x {v}")
Send($"x {items.Aggregate(...)}") Build via ValueStringBuilder, pass span

For lowercase output, use the :L format specifier instead of value.ToString().ToLowerInvariant():

mob.SendMessage($"You earned a {rank:L} trophy!");          // "gold" not "Gold"

Why: These methods are called constantly during gameplay (every chat line, every system message, every gump label, every tooltip). The handler overload exists specifically to eliminate per-call string allocation. Each anti-pattern leaks one or more strings per call.

Severity: WARNING. Flag and ask before fixing — some patterns (e.g., reused locals across multiple call sites) are intentional and shouldn't be inlined.

See: dev-docs/string-handling.md § "Interpolation Anti-Patterns" for the full reference with detailed before/after examples.

19. No InvalidateProperties From Inside GetProperties

Check: Any property read by a GetProperties override — including through helpers — must be a pure read. Flag getters that call InvalidateProperties() (or a wrapper like Invalidate()) as a side effect. Bad: a Rank getter that lazily recomputes and then calls Invalidate(); reading it from GetProperties re-enters the build. Good: invalidate in the setter that actually changes the value, or defer with Timer.DelayCall(InvalidateProperties). Why: InvalidateProperties() rebuilds the list in place (Reset() + rebuild). Reset() returns the pooled interpolation buffer — which the compiler rents for the whole $"..." expression, so every hole is evaluated while it is live — and rewinds the packet cursor. Re-entering mid-build throws ArgumentNullException (parameter "array") out of GetProperties from a line unrelated to the offending getter, or silently corrupts the tooltip. The engine refuses and logs an error, and DEBUG throws, so this shows up as a crash in development. Note: Lazy recomputation inside a getter is fine. It is the notification that must not happen there.

See: dev-docs/property-lists.md § "Never Invalidate From Inside GetProperties".

20. Tick-Count Math Must Be Wraparound-Safe

Check: Every comparison between Core.TickCount / Core.GetTimestamp() values (or fields derived from them — names like *Until, *At, *Next*, deadline) must be in subtraction form. Flag direct comparisons, zero/sign sentinels, and deadline fields left at their zero default. Bad: if (Core.TickCount < _deadline); if (_lastEventAt > 0) as "has happened"; private static long _deadline; compared before being seeded from a real tick. Good: if (Core.TickCount - _deadline < 0); a separate bool for "has happened"; seeding deadline fields from the first observed timestamp. Why: On some hypervisors — Google Cloud specifically — the VM receives a pass-through of the host's never-resetting counter. Tick counts are NOT zero at process start, NOT zero at OS boot, can be enormous from the first read, and can wrap negative. Direct comparisons and sign sentinels then fail only on those hosts, after long host uptimes — the least reproducible bug class there is. Windows has not shown this in testing; Linux has, in production. Subtraction of two ticks wraps correctly in two's complement. Note: DateTime/DateTimeOffset comparisons are unaffected; this applies only to the monotonic tick domain.

See: dev-docs/tick-counts.md for the full rules and review checklist.

Severity Levels

  • ERROR: Rules 3, 9, 10, 13, 19, 20 (will cause bugs, build failures, or client-side leaks)
  • WARNING: Rules 1 (Tier 3 LINQ), 2, 4, 5, 6, 7, 8, 12, 14, 15, 17 (performance/convention issues)
  • INFO: Rules 1 (Tier 2 LINQ on warm paths — note it but don't flag as violation), 16 (switch patterns — suggest but don't flag)
  • ASK: Rule 11 (need user input)

How to Report

When you find violations, report them as:

[AUDIT] {SEVERITY}: {Description}
  File: {path}:{line}
  Suggestion: {fix}

Do NOT silently fix issues. Always flag and ask.

See Also

  • dev-docs/code-standards.md - Full coding standards documentation
  • dev-docs/claude-skills/modernuo-serialization.md - Serialization rules
  • dev-docs/claude-skills/modernuo-timers.md - Timer cleanup rules
  • dev-docs/claude-skills/modernuo-threading.md - Threading model details
  • dev-docs/claude-skills/modernuo-property-lists.md - PropertyList interpolation rules