## Problem
`RunEventLoop` span through its body regardless of whether there was anything to do — ~10% of a desktop core for an empty shard, and ~70% of a core on a 3 vCPU VPS. A process that never idles is exactly what burstable vCPU plans throttle, which is how this surfaced: lag spikes that went away when the operator bought more cores. The spin also denied the GC its natural pause points, so memory climbed until a world save forced a collection — alarming in task manager, harmless in practice, and a recurring source of "is my server leaking?" reports.
## Result
Windows desktop, real world of **190,728 items / 33,158 mobiles**, no players, saves and prebake off, three consecutive runs:
| | Legacy spin | Idle sleeping |
|---|---|---|
| **CPU** | 10.42 – 10.50% of one core | **0.78 – 1.00%** |
| **Tick lag** (peak/15s) | 4–10 ms | 5–11 ms |
**~10× less CPU with tick lag unchanged** — the CPU came free rather than being traded for latency. Slower hosts gain proportionally more. Spin mode (`server.eventLoopIdleWaitMs=0`) independently gained **7× the iterations per core** (1.19M → 8.3M cycles/sec) from the ring's AcceptEx rework.
## How
The loop blocks in `NetState.WaitForCompletion` whenever every queue it drains is empty (all the drains are bounded, so leftovers keep it awake). Receive completions, new connections, and cross-thread `LoopContext.Post` (via the ring's sticky `Wake()`) are all in the wait set, so sleeping adds no latency to any of them. Only timer-driven logic sees wheel lag, bounded by the idle wait.
**Health is measured at the only place sleeping can cause harm.** A sleep is bounded by the time to the next wheel turn, so a correctly honoured sleep can never miss a deadline — the only failure mode is the host returning the wait late. That overshoot is measured on every sleep (one extra timestamp read; production's entire accounting cost), and an escalating backoff suspends sleeping when it persists. By construction, server work — saves, heavy staff commands, deep timer callbacks — cannot trip it, so the warning means exactly one thing: *the host is not scheduling the process promptly*, with two known remedies (dedicated CPU, or `=0`). Hosts with no high-resolution wait mechanism at all are detected once at startup and spin instead.
**CPS is removed.** `Core.CyclesPerSecond`/`AverageCPS` measured nothing actionable before and became actively misleading once the loop sleeps (the rate is set by the sleep, not by shard health). The admin gump's Performance page now shows the verdict instead: `Healthy` / `Sleep suspended (host)` / `Spinning (configured)`.
## Configuration
| Setting | Default | Meaning |
|---|---|---|
| `server.eventLoopIdleWaitMs` | `2` | Longest idle block. Measured across 1/2/4/8 ms, 2 is where the trade stops being free. `0` = never sleep: ~98% of a core, zero scheduling overhead — for large shards on dedicated CPU. |
| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before sleeping backs off. Raise for jittery hosts; very high disables the backoff. |
## Diagnostics (compiled out by default)
`dotnet build -p:EventLoopProfiling=true` compiles in `EventLoopProfiler` — every hook is `[Conditional("EVENT_LOOP_PROFILING")]`, so normal builds contain zero profiling IL. The profiling build decomposes each second of wall time into **work (per loop phase) / sleep / GC pause / stolen residual**, keeps ~15 minutes of history in a ring buffer, and the `[LoopStats` command prints the last minute and dumps the full history to CSV. `dev-docs/debugging-event-loop.md` is the diagnosis guide (for humans and AI): what production already tells you, when to flip the profiling build, the signature table for host-steal vs deep-processing vs GC vs wake bugs, why dotnet-trace comes last, and the GC/RAM "leak" misconception.
## Verification
- 815 Server.Tests green; both build configurations compile.
- Docker echo harness green on epoll and io_uring (ping-pong mode); kqueue verified manually on an M1 Max.
- A/B measurements and per-change numbers: `measure/event-loop` branch.
## Notes
The full measurement harness and vendored ring sources used to develop this live on the [`measure/event-loop`](https://github.com/modernuo/ModernUO/tree/measure/event-loop) branch, kept for future loop work.
15 KiB
| name | description |
|---|---|
| modernuo-code-audit | Auto-trigger whenever writing or modifying .cs files under Projects/. Audits code for ModernUO convention violations. Warnings only - flag issues and ask before fixing. |
ModernUO Code Audit
When This Activates
- Any time you write, edit, or modify a
.csfile underProjects/ - After generating code snippets for the user
- During code review
Audit Rules (Warnings Only)
Flag these issues but do NOT auto-fix. Ask the user before making changes.
1. LINQ: Know What's Optimized (.NET 10)
Not all LINQ is banned. .NET 10 JIT/PGO eliminates overhead for specific patterns. Anything not listed below is still forbidden on hot paths.
Tier 1 — Zero-cost (use freely on hot paths):
foreachoverIEnumerable<T>backed byT[],List<T>,Stack<T>,Queue<T>— PGO devirtualizes the enumerator, zero heap allocation.Contains()after a preceding LINQ operator (.Distinct(),.OrderBy(),.Reverse(),.Union(),.Intersect(),.Except(),.Concat(),.SelectMany(),.Where().Select(),.Skip(),.Take(),.OfType(),.Cast(),.Shuffle()) — LINQ has ~30 specialized overrides that skip the intermediate work (no sort, no HashSet, no buffering).Count()on sized collections (ICollection<T>, or afterRange/Repeat/Skip/Take/Append) — O(1) property access, no enumeration.OrderBy().First()/.OrderByDescending().First()/.OrderBy().Last()— O(N) min/max scan, no sort performed.Shuffle().Take(n)— reservoir sampling, single pass, O(n) memoryEnumerable.Range()/Enumerable.Sequence()followed by.Count(),.Contains(),.ToArray(),.ToList(),.ElementAt(),.Last()— arithmetic, not enumeration
Tier 2 — Low overhead (acceptable on warm paths, benchmark if critical):
.Skip(n).Take(m).ToArray()onT[]/List<T>— vectorizedSpan<T>.CopyTo(still allocates output).LeftJoin()/.RightJoin()— ~2x faster than manualGroupJoin+SelectMany+DefaultIfEmpty.Where(predicate)onT[]/List<T>—WhereIteratorstill heap-allocates, but enumeration is PGO-optimized. Manualforeach+ifis still faster for true hot paths.
Tier 3 — Still forbidden on hot paths (write manual code):
.Select(f).Where(p)(this order — each intermediate iterator allocates).GroupBy(),.ToDictionary(),.ToHashSet(),.ToLookup()(always allocate internal structures).Aggregate()(delegate overhead per element).Sum()/.Min()/.Max()onfloat/double(no SIMD in LINQ on ARM).SelectMany()when iterating results (not.Contains()) — multiple enumerator allocations.Zip()when iterating — enumerator allocations- Any LINQ over
IAsyncEnumerable<T>— no PGO/escape analysis - Long chains like
.Where().Select().OrderBy().Take()— each step allocates an iterator
Prerequisites: .NET 10, tiered compilation + Dynamic PGO enabled (default). Tier 1 optimizations require ~30+ calls for JIT warmup.
Quick decision: If the exact pattern is in Tier 1 → use it. If it's in Tier 2 → acceptable unless profiling shows it's a bottleneck. If it's anything else → manual for/foreach + PooledRefList<T>.
2. No Console.WriteLine
Bad: Console.WriteLine(...), Console.Write(...)
Good: private static readonly ILogger logger = LogFactory.GetLogger(typeof(MyClass)); then logger.Information(...), logger.Warning(...), logger.Error(...)
Requires: using Server.Logging;
3. No Concurrency Primitives in Game Code
Bad: ConcurrentDictionary, ConcurrentQueue, ConcurrentBag, volatile, lock(...), Mutex, Semaphore, Monitor, Interlocked, ReaderWriterLock
Why: Server is single-threaded. These add overhead for no benefit.
Instead: Use regular Dictionary<K,V>, List<T>, plain fields.
4. Never Iterate World.Mobiles or World.Items Directly
Bad: foreach (var m in World.Mobiles.Values), World.Items.Values.Where(...)
Good: map.GetMobilesInBounds<T>(bounds), map.GetMobilesInRange<T>(point, range), map.GetItemsInRange<T>(point, range)
Why: Full world iteration is O(n) over all entities. Spatial queries use sector indexing.
5. Clean Up References in OnDelete/OnAfterDelete
Check: Classes with Item or Mobile references should clean them in OnDelete() or OnAfterDelete().
Pattern:
public override void OnAfterDelete()
{
_someReference = null;
base.OnAfterDelete();
}
6. Cancel Timers in OnDelete/OnAfterDelete
Check: Any class with TimerExecutionToken or Timer fields must cancel them on deletion.
Pattern:
public override void OnAfterDelete()
{
_timerToken.Cancel(); // For TimerExecutionToken
_timer?.Stop(); // For Timer references
_timer = null;
base.OnAfterDelete();
}
7. Use STArrayPool, Not ArrayPool
Bad: ArrayPool<T>.Shared.Rent(...) in game logic
Good: STArrayPool<T>.Shared.Rent(...) in game logic
Why: STArrayPool is single-threaded optimized (no locks). Use ArrayPool only in explicitly multi-threaded code.
Also: Always return rented arrays in a finally block.
8. No new List in Hot Paths
Bad: var list = new List<Mobile>(); in frequently-called methods
Good: using var list = PooledRefList<Mobile>.Create();
Why: PooledRefList uses pooled arrays, zero GC pressure. It's a ref struct (stack-allocated).
9. Serialization Class Requirements
Check: Classes with [SerializationGenerator] MUST be partial.
Check: [Constructible] on parameterless constructors for items/mobiles.
Check: TimerExecutionToken fields must NOT have [SerializableField].
Check: Use using ModernUO.Serialization; when using serialization attributes.
10. No Task.Run or new Thread
Bad: Task.Run(...), new Thread(...), ThreadPool.QueueUserWorkItem(...) in game code
Why: Game logic runs on the single-threaded event loop. Background threads cause race conditions.
Exception: Server infrastructure code (Projects/Server/Main.cs, World saves) may use threading.
11. Never Assume Era
Check: If code uses era-conditional logic (Core.AOS, Core.SE, etc.) and the user hasn't specified a target era, ASK which expansion to target.
Why: Different eras have dramatically different mechanics.
12. Naming Conventions
Check: _camelCase for private fields, PascalCase for properties/methods/classes.
Note: Legacy code may use m_ prefix -- don't flag existing m_ fields but use _ for new code.
13. No Empty Gumps
Check: Any gump (legacy Gump constructor, or BuildLayout) must not have a code path that produces zero visual elements (no AddBackground, no AddPage with content, etc.).
Why: The client has no way to close an empty gump — no close button, no right-click dismiss. This leaks a gump slot on both client and server until relog.
Common cause: Early return in a constructor or BuildLayout when prerequisites aren't met.
Fix: Use a static DisplayTo(Mobile from) method that validates prerequisites before constructing the gump. Make the constructor private. See Projects/UOContent/Gumps/Go/GoGump.cs for the canonical pattern.
14. PropertyList String Literals Must Be Holes
Check: In any IPropertyList.Add() interpolated string, string constants must be wrapped as holes {"text"}, not bare literals.
Bad: list.Add(1060658, $"Chances\t{_charges}"); — "Chances" becomes a delimiter, not an argument.
Good: list.Add(1060658, $"{"Chances"}\t{_charges}"); — "Chances" is an argument.
Why: The handler treats bare text as delimiters and {} contents as arguments. The property list system is used beyond the game client (e.g., web rendering) which must distinguish arguments from delimiters. Only \t should be a bare literal.
Also: If you don't know the text for a cliloc number, see Projects/Server/Localization/Localization.cs LoadClilocs() to learn the binary format, and ask the user where their cliloc.enu file is.
15. Braces Required on All Control Flow
Check: ALL if, else, for, foreach, while, do, switch statements must have braces, even for single-line bodies.
Bad:
if (condition)
DoSomething();
Good:
if (condition)
{
DoSomething();
}
Why: Reduces merge conflicts and diff sizes.
16. Prefer Switch Expressions and Switch-When Patterns
Check: Where a chain of if/else if maps inputs to outputs, prefer a switch expression. Where pattern matching with guards improves clarity, prefer switch-when.
Bad:
if (type == GemType.StarSapphire) return "star sapphire";
else if (type == GemType.Emerald) return "emerald";
else return "gem";
Good:
return type switch
{
GemType.StarSapphire => "star sapphire",
GemType.Emerald => "emerald",
_ => "gem"
};
Why: Switch expressions enable JIT/PGO optimization and improve readability. Exception: Skip if the switch would be unreadable or the code is on a cold path.
17. Interpolation Anti-Patterns (handler-aware APIs)
Context: Many ModernUO APIs accept ref RawInterpolatedStringHandler (Mobile.SendMessage/Say/Emote/etc., Item.Public/Local/NonlocalOverheadMessage/SendLocalizedMessageTo/SendMessageTo, IPropertyList.Add, SpanWriter.WriteAscii/WriteLatin1, gump AddLabel/AddHtml/AddHtmlLocalized, Html.Center/Color/Right). The handler overload renders the interpolation directly into a pooled buffer with zero string allocation — but only when the call-site argument is a $"..." literal directly in the parameter slot.
Check: Flag any of the following patterns when the call target is one of those handler-aware APIs. The handler overload is silently bypassed and a string is allocated per call.
| Pattern | Fix |
|---|---|
Send(cond ? $"a" : $"b") |
if/else with two calls |
Send(thing switch { 1 => $"a", _ => $"b" }) |
switch statement, call per arm |
var s = $"foo {x}"; Send(s); (single-use) |
Inline at call site |
Send($"x {value.ToString()}") |
Drop .ToString() — handler formats directly |
Send($"x {td.String()}") |
Drop .String() — pass td directly |
Send($"x {a + b}") (string concat) |
Multiple holes: Send($"x {a}{b}") |
Send(string.Format("x {0}", v)) |
Send($"x {v}") |
Send($"x {items.Aggregate(...)}") |
Build via ValueStringBuilder, pass span |
For lowercase output, use the :L format specifier instead of value.ToString().ToLowerInvariant():
mob.SendMessage($"You earned a {rank:L} trophy!"); // "gold" not "Gold"
Why: These methods are called constantly during gameplay (every chat line, every system message, every gump label, every tooltip). The handler overload exists specifically to eliminate per-call string allocation. Each anti-pattern leaks one or more strings per call.
Severity: WARNING. Flag and ask before fixing — some patterns (e.g., reused locals across multiple call sites) are intentional and shouldn't be inlined.
See: dev-docs/string-handling.md § "Interpolation Anti-Patterns" for the full reference with detailed before/after examples.
19. No InvalidateProperties From Inside GetProperties
Check: Any property read by a GetProperties override — including through helpers — must be a pure read. Flag getters that call InvalidateProperties() (or a wrapper like Invalidate()) as a side effect.
Bad: a Rank getter that lazily recomputes and then calls Invalidate(); reading it from GetProperties re-enters the build.
Good: invalidate in the setter that actually changes the value, or defer with Timer.DelayCall(InvalidateProperties).
Why: InvalidateProperties() rebuilds the list in place (Reset() + rebuild). Reset() returns the pooled interpolation buffer — which the compiler rents for the whole $"..." expression, so every hole is evaluated while it is live — and rewinds the packet cursor. Re-entering mid-build throws ArgumentNullException (parameter "array") out of GetProperties from a line unrelated to the offending getter, or silently corrupts the tooltip. The engine refuses and logs an error, and DEBUG throws, so this shows up as a crash in development.
Note: Lazy recomputation inside a getter is fine. It is the notification that must not happen there.
See: dev-docs/property-lists.md § "Never Invalidate From Inside GetProperties".
20. Tick-Count Math Must Be Wraparound-Safe
Check: Every comparison between Core.TickCount / Core.GetTimestamp() values (or fields
derived from them — names like *Until, *At, *Next*, deadline) must be in subtraction form.
Flag direct comparisons, zero/sign sentinels, and deadline fields left at their zero default.
Bad: if (Core.TickCount < _deadline); if (_lastEventAt > 0) as "has happened";
private static long _deadline; compared before being seeded from a real tick.
Good: if (Core.TickCount - _deadline < 0); a separate bool for "has happened"; seeding
deadline fields from the first observed timestamp.
Why: On some hypervisors — Google Cloud specifically — the VM receives a pass-through of the
host's never-resetting counter. Tick counts are NOT zero at process start, NOT zero at OS boot,
can be enormous from the first read, and can wrap negative. Direct comparisons and sign sentinels
then fail only on those hosts, after long host uptimes — the least reproducible bug class there
is. Windows has not shown this in testing; Linux has, in production. Subtraction of two ticks
wraps correctly in two's complement.
Note: DateTime/DateTimeOffset comparisons are unaffected; this applies only to the
monotonic tick domain.
See: dev-docs/tick-counts.md for the full rules and review checklist.
Severity Levels
- ERROR: Rules 3, 9, 10, 13, 19, 20 (will cause bugs, build failures, or client-side leaks)
- WARNING: Rules 1 (Tier 3 LINQ), 2, 4, 5, 6, 7, 8, 12, 14, 15, 17 (performance/convention issues)
- INFO: Rules 1 (Tier 2 LINQ on warm paths — note it but don't flag as violation), 16 (switch patterns — suggest but don't flag)
- ASK: Rule 11 (need user input)
How to Report
When you find violations, report them as:
[AUDIT] {SEVERITY}: {Description}
File: {path}:{line}
Suggestion: {fix}
Do NOT silently fix issues. Always flag and ask.
See Also
dev-docs/code-standards.md- Full coding standards documentationdev-docs/claude-skills/modernuo-serialization.md- Serialization rulesdev-docs/claude-skills/modernuo-timers.md- Timer cleanup rulesdev-docs/claude-skills/modernuo-threading.md- Threading model detailsdev-docs/claude-skills/modernuo-property-lists.md- PropertyList interpolation rules