perf(core): sleep the event loop when idle

The loop span through its body regardless of whether there was anything
to do -- ~10% of a desktop core for an empty shard, ~70% of a small VPS
core, and a process that never idles is exactly what burstable vCPU
plans throttle.

The loop now blocks in NetState.WaitForCompletion whenever every queue
it drains is empty, waking on the next timer tick or the moment work
arrives. Receive completions, new connections, and cross-thread
LoopContext.Post (via IORingGroup 1.0.10's sticky Wake) are all in the
wait set, so sleeping adds no latency to any of them; only timer-driven
logic sees wheel lag, bounded by server.eventLoopIdleWaitMs (default
2ms, 0 = never sleep).

Measured on a real world of 190k items / 33k mobiles: 10.4% of a core
to 0.8-1.0%, with peak tick lag unchanged. Spin mode independently
gained 7x the iterations per core from the ring's AcceptEx rework.
Sleeping also gives the GC natural pause points, which the old spin
loop denied it -- memory no longer climbs until a save forces a
collection.

A sleep is bounded by the next wheel turn, so a correctly honoured
sleep can never miss a deadline; the only way sleeping harms the wheel
is the host returning the wait late. That overshoot is measured on
every sleep, and an escalating backoff (server.lateWakeThreshold)
suspends sleeping when it persists -- server work like saves or heavy
commands cannot trip it by construction. Hosts without high-resolution
waits are detected once at startup and spin instead. The admin gump
shows the verdict instead of the now-meaningless CPS figure, which is
removed.

Time accounting for diagnosis is compiled out of normal builds: build
with -p:EventLoopProfiling=true to enable EventLoopProfiler (per-phase
wall time, sleep overshoot, GC pauses, stolen-time residual, ~15min
ring buffer) and the [LoopStats command with CSV dump. See
dev-docs/debugging-event-loop.md for the diagnosis funnel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dev-docs/claude-skills/modernuo-code-audit.md

@@ -200,8 +200,27 @@ mob.SendMessage($"You earned a {rank:L} trophy!");          // "gold" not "Gold"
 
 **See**: `dev-docs/property-lists.md` § "Never Invalidate From Inside `GetProperties`".
 
+### 20. Tick-Count Math Must Be Wraparound-Safe
+**Check**: Every comparison between `Core.TickCount` / `Core.GetTimestamp()` values (or fields
+derived from them — names like `*Until`, `*At`, `*Next*`, `deadline`) must be in subtraction form.
+Flag direct comparisons, zero/sign sentinels, and deadline fields left at their zero default.
+**Bad**: `if (Core.TickCount < _deadline)`; `if (_lastEventAt > 0)` as "has happened";
+`private static long _deadline;` compared before being seeded from a real tick.
+**Good**: `if (Core.TickCount - _deadline < 0)`; a separate `bool` for "has happened"; seeding
+deadline fields from the first observed timestamp.
+**Why**: On some hypervisors — Google Cloud specifically — the VM receives a pass-through of the
+host's never-resetting counter. Tick counts are NOT zero at process start, NOT zero at OS boot,
+can be enormous from the first read, and can wrap negative. Direct comparisons and sign sentinels
+then fail only on those hosts, after long host uptimes — the least reproducible bug class there
+is. Windows has not shown this in testing; Linux has, in production. Subtraction of two ticks
+wraps correctly in two's complement.
+**Note**: `DateTime`/`DateTimeOffset` comparisons are unaffected; this applies only to the
+monotonic tick domain.
+
+**See**: `dev-docs/tick-counts.md` for the full rules and review checklist.
+
 ## Severity Levels
-- **ERROR**: Rules 3, 9, 10, 13, 19 (will cause bugs, build failures, or client-side leaks)
+- **ERROR**: Rules 3, 9, 10, 13, 19, 20 (will cause bugs, build failures, or client-side leaks)
 - **WARNING**: Rules 1 (Tier 3 LINQ), 2, 4, 5, 6, 7, 8, 12, 14, 15, 17 (performance/convention issues)
 - **INFO**: Rules 1 (Tier 2 LINQ on warm paths — note it but don't flag as violation), 16 (switch patterns — suggest but don't flag)
 - **ASK**: Rule 11 (need user input)

dev-docs/debugging-event-loop.md

@@ -0,0 +1,115 @@
+# Debugging Event Loop Performance
+
+How to diagnose "the server feels slow" — written for both humans and AI assistants. Follow the
+funnel in order; most incidents resolve before the last step. Do not start with dotnet-trace.
+
+## The model
+
+Every second of the main thread's wall time goes to exactly one of four places:
+
+1. **Work** — the loop's phases: mobile deltas, item deltas, timer callbacks (`Timer.Slice`),
+   network processing (`NetState.Slice`), posted tasks (`LoopContext`).
+2. **Sleep** — idle blocking in `NetState.WaitForCompletion`, bounded by the next timer tick and
+   `server.eventLoopIdleWaitMs`.
+3. **GC pauses** — land inside whichever phase (or sleep) was running.
+4. **Stolen** — the host ran something else: hypervisor scheduling, noisy neighbors, CPU credit
+   throttling.
+
+A sleep is bounded by the time to the next wheel turn, so **a correctly honoured sleep can never
+cost a deadline**. The only way sleeping harms the game is the wait *returning late* — that is
+stolen time, and the server measures it directly on every sleep.
+
+## Step 0 — Read what production already tells you
+
+No build changes needed. Three signals exist, all actionable:
+
+| Signal | Meaning | Action |
+|---|---|---|
+| Startup error: *host cannot honour short waits* | No high-resolution timer and `timeBeginPeriod` failed. Very old or unusual Windows. | Nothing is wrong with the server; it spins and uses a full core. Upgrade the OS or accept the core. |
+| Warning: *host returned a Nms idle wait late* + sleeping suspended | The OS did not reschedule the process promptly after a 1–2ms wait. Shared/burstable vCPU signature. | Move to dedicated CPU, or set `server.eventLoopIdleWaitMs=0` to spin permanently. This is a **host** problem — no amount of server-side change fixes it. |
+| Admin gump → Performance → *Event Loop* | `Healthy` / `Sleep suspended (host)` / `Spinning (configured)` | Same as above. |
+
+If none of these fired and the shard still feels laggy, the cause is work, GC, or something a
+boot-time signal cannot see. Continue.
+
+## Step 1 — Flip the profiling build
+
+```
+dotnet build -p:EventLoopProfiling=true
+```
+
+This compiles in `EventLoopProfiler` (Server) and the `[LoopStats` command (UOContent). Without
+the flag every hook call site is removed by the compiler (`[Conditional]`), so there is nothing to
+"turn off" in normal builds and no cost to leave the hooks in the code. The profiling build's own
+overhead is a handful of timestamp reads per iteration — small enough to run for days while
+hunting an intermittent problem.
+
+**Capture a baseline first.** Run `[LoopStats` while the shard feels *fine* and keep the CSV. The
+profiler also keeps ~15 minutes of history in memory, so if the problem is episodic you can wait
+for an episode and the good minutes on either side are already recorded. Numbers without a
+baseline are how RunUO's profiler became useless — always compare bad minutes to good minutes on
+the same box, build, and world.
+
+## Step 2 — Read the decomposition
+
+`[LoopStats` prints the last minute and writes the full history CSV (one row per second). Match
+the shape against these signatures:
+
+| Signature | Diagnosis | Next step |
+|---|---|---|
+| One phase consistently hot (e.g. `TimerSlice` 40%/s) | Deep processing in that subsystem | Step 3 — find the culprit in that phase |
+| All phases near zero, `stolen` high, `lateWakes` > 0 | Host is stealing CPU | Host problem; see step 0 actions |
+| `gcPauseMs` high, gen2 counts rising | GC pressure — something is allocating heavily | Step 3 on the allocating phase, or dotnet-counters for alloc rate |
+| Iterations ≫ sleeps while shard is idle | The loop is not sleeping: a queue never drains or a wake storm | Check `IsIdle` inputs; a stuck signal in the ring is the historical example |
+| Sleeps ≈ iterations, each sleep ~0ms | Spurious wake storm | Ring backend issue; count `wakesIssued` vs actual cross-thread posts |
+| Everything normal, complaint persists | Not the event loop | Look at the network path, client, or DB/save timing |
+
+**Wheel lag vs player lag:** `wheelLagMaxMs` is how late timer callbacks fired. Receives are
+handled the moment they arrive (they wake the loop), so player-felt lag with a clean wheel points
+away from the loop entirely.
+
+## Step 3 — Find the culprit inside a hot phase
+
+Add a temporary culprit hook rather than reaching for a tracer. The pattern: same
+`[Conditional("EVENT_LOOP_PROFILING")]` attribute, own file or the profiler file, record only the
+worst offender per second (identity + duration), never a per-event log. Examples:
+
+- `TimerSlice` hot → time each timer callback, keep the max and its `timer.ToString()`.
+- `NetworkSlice` hot → time packet handlers by packet id, keep the max.
+- GC pressure → `dotnet-counters monitor --counters System.Runtime` for alloc rate first; it is
+  cheap and often names the culprit generation without a trace.
+
+Keep the hook after the hunt if it earns its cost in the profiling build; delete it otherwise.
+
+## Step 4 — dotnet-trace, last and targeted
+
+Only when a hot phase resists the culprit hook. Know the costs: EventPipe visibly slows the
+process (worst exactly when things are already bad) and adds artifacts to the trace — on small
+vCPU hosts the tracer's own threads appear as hotspots and Rider/PerfView hotspot views can
+mislead. Mitigate by being narrow:
+
+- Trace the specific minutes the decomposition flagged, not "a while".
+- `dotnet-trace collect --profile cpu-sampling --duration 00:00:30` is usually enough.
+- Compare against a trace of a good minute (same rule as step 1: no baseline, no conclusions).
+
+## The RAM / GC misconception (read before declaring a leak)
+
+ModernUO allocates very little, and the GC collects opportunistically — mostly during idle sleeps
+and world saves. Under a spinning loop (`eventLoopIdleWaitMs=0`, or the pre-2026 default) the GC
+may find **no** natural pause point: memory climbs to a large fraction of physical RAM, a forced
+collection eventually drops part of it, and fragmentation keeps the baseline permanently above
+where it started. Task manager shows alarming numbers; the in-game numbers do not. **Performance
+is unaffected — this is lazy collection working as designed, not a leak.** Idle sleeping largely
+removes the effect because every sleep is a natural GC opportunity. Before investigating "a leak":
+check `gen0/1/2` and `gcPauseMs` in the decomposition, and compare working set *after a world
+save*, which forces the collection the spin loop never allowed.
+
+## Rules of thumb
+
+- Never trade always-on profiling for the numbers. Production carries one timestamp per sleep and
+  nothing else; everything heavier lives behind the build flag or on the `measure/event-loop`
+  branch (full harness, A/B scripts, vendored ring experiments).
+- One decomposition chart beats a thousand log lines. Resist adding warnings the reader cannot
+  act on; the three production signals are deliberate.
+- When filing or reporting: attach the baseline CSV and the episode CSV. Relative statements
+  ("TimerSlice went from 4% to 61% during the episode") are the useful form.

dev-docs/server-requirements.md

@@ -0,0 +1,120 @@
+# Server Requirements
+
+Hardware guidance for running a ModernUO shard.
+
+## Tiers
+
+| Use | vCPU | RAM | Storage |
+|---|---|---|---|
+| Development / test | 2 **dedicated** | 2 GB | SSD |
+| Small live shard (< 50 concurrent) | 4 dedicated | 4 GB | NVMe |
+| Medium (50–200) | 4–8 | 8 GB | NVMe |
+| Large (200+) | 8+, high clock | 16 GB+ | NVMe |
+
+These are starting points. Save size drives RAM more than player count does, and single-thread
+clock speed drives tick latency more than core count does. Both are explained below.
+
+## Dedicated vCPU, not burstable
+
+This matters more than any other line on this page.
+
+Budget VPS plans sold as "2 vCPU" are frequently shared or burstable: you get a CPU credit balance
+or a cgroup quota, and once it is exhausted the hypervisor throttles you. Throttling shows up in
+game as periodic freezes that correlate with nothing in your logs, and it is the single most common
+cause of "ModernUO is laggy on my $3/month VPS".
+
+Symptoms worth checking before blaming the server:
+
+- Steal time above ~1% (`top`, the `%st` column on Linux)
+- Lag that disappears when you move to a larger plan with the same core count
+- Tick lag spikes with no matching CPU spike in the process itself
+
+## Cores
+
+Game logic is **single-threaded**. Every mobile, item, timer, and packet handler runs on one
+thread, so a shard's headroom is bounded by how fast one core is. Two fast cores beat four slow
+ones.
+
+Cores beyond the first are used by:
+
+- **World saves.** `world.useMultithreadedSaves` (default on) spins up `ProcessorCount - 1`
+  serialization workers plus one inline on the main thread. On a 2-core box that is one worker; on
+  a 2-core box with a large world, consider setting it to `false` so saves do not contend with the
+  loop.
+- **The .NET runtime.** Tiered JIT compilation (heaviest in the first minutes after boot) and
+  background GC.
+- **Everything else on the machine**, including your OS and, on Windows, antivirus.
+
+Since ModernUO 2026 the loop sleeps when idle, so an empty shard costs roughly 1% of a core rather
+than spinning. That change disproportionately helps small hosts.
+
+## Memory
+
+Three things dominate, and only one of them scales with players.
+
+**World size.** A world of ~190,000 items and ~33,000 mobiles loads in about a second and is not
+itself large. Items and mobiles are the cheap part.
+
+**Saves.** Each serialization worker pre-allocates a heap sized to its share of the last save, at
+roughly 1.25× total save size, and those buffers are retained afterwards. A 400 MB save therefore
+implies about 500 MB of resident serialization heap on top of the live world. **This is the reason
+1 GB hosts are not viable for a real shard**, even though an empty one boots fine.
+
+**Map residency.** `TileMatrix` reads map blocks from disk on demand and caches them permanently —
+there is no eviction. Memory climbs toward full-facet residency as players explore. Felucca's land
+tiles alone are around 117 MB, and statics are larger.
+
+Optional systems can add substantially more. The pathfinding prebake
+(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
+small hosts.
+
+Network buffers are minor by comparison: 64 KB receive plus a configurable 256 KB send
+(`network.sendBufferSize`) per connection, so 100 players is roughly 32 MB.
+
+ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
+Server GC on a 2-core box.
+
+## Storage
+
+Saves are write-heavy bursts. Cheap network-attached storage with throttled IOPS will stall the
+save path, and `World.WaitForWriteCompletion` blocks the loop at shutdown. Use local NVMe or SSD.
+
+Budget disk for: the world save, plus archives and backups if `autoArchive` is enabled (retention
+defaults keep 24 hourly, 30 daily, and 12 monthly copies), plus the pathfinding cache if enabled.
+
+## Operating systems
+
+See the README for the full supported list. Two things are worth calling out:
+
+- **Windows Server 2012 R2 and 2016 sleep via a raised timer resolution.** Sleeping for a couple
+  of milliseconds prefers a high-resolution waitable timer, which requires Windows 10 1803 /
+  Server 2019. On older versions the ring falls back to `timeBeginPeriod(1)`, which raises the
+  system timer resolution to 1 ms so the plain wait timeout is accurate enough. The trade-off is a
+  higher interrupt rate (system-wide on those versions) — an acceptable price on a dedicated game
+  server, and the reason the high-resolution timer is preferred where it exists.
+
+  Only if *both* mechanisms fail does the server detect it at startup, log it, and spin instead —
+  the same behaviour as setting `server.eventLoopIdleWaitMs` to 0: a full core at idle, and zero
+  missed deadlines. A host that claims short waits but cannot deliver them is caught at runtime by
+  the adaptive backoff.
+- **Linux kernel 6.1** or newer (Debian 12 and equivalents). io_uring is used where available, with
+  automatic epoll fallback.
+
+## Tuning for a small host
+
+| Setting | Default | Why change it |
+|---|---|---|
+| `server.eventLoopIdleWaitMs` | `2` | `0` never sleeps: ~98% of one core, but zero skipped timer slots and zero lag. The choice for a large shard on dedicated CPU that would rather spend a core than risk a late wake. Above `2` the wheel starts losing slots. |
+| `server.lateWakeThreshold` | `1` | Idle waits the host may return a full tick late, per second, before idle sleeping backs off. Raise on a jittery host; set very high to disable the backoff. |
+| `world.useMultithreadedSaves` | `true` | Set `false` on 2-core hosts so saves do not contend with the game loop. |
+| `pathfinding.prebakeMaps` | varies | Leave off on memory-constrained hosts; it peaks above 1 GB while baking. |
+| `network.sendBufferSize` | 256 KB | Lower it if you are memory-bound with many connections. |
+| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
+
+## Am I undersized?
+
+Watch the log. The server warns when the host returns idle waits late and suspends idle sleeping,
+and says so at startup if the host cannot honour short waits at all. Those warnings mean the host
+is not scheduling the process promptly — typical of burstable or shared vCPU plans — and no
+server-side change fixes that. For anything deeper, see
+[debugging-event-loop.md](debugging-event-loop.md).

dev-docs/tick-counts.md

@@ -0,0 +1,55 @@
+# Tick Counts: Overflow and Huge Starting Values
+
+Rules for any code that compares `Core.TickCount` / `Core.GetTimestamp()` values. Getting this
+wrong produces bugs that only appear on specific cloud hosts after long host uptimes — the worst
+kind to reproduce.
+
+## Why this matters (the Linux/cloud problem)
+
+`Core.GetTimestamp()` is built on `Stopwatch.GetTimestamp()`, which on Linux reads the kernel's
+monotonic clock — and on some hypervisors, notably **Google Cloud**, the VM receives a
+**pass-through of the host's never-resetting counter**. The tick count is *not* zero when the
+process starts and *not* zero when the operating system booted; it is however long the physical
+host has been up, which can be months or years. We have been burned by this in production.
+
+Consequences:
+
+- Raw values are enormous from the first read. Arithmetic that would "never overflow in 292
+  years" of process uptime can overflow immediately (`Core.GetTimestamp()`'s `UInt128`
+  conversion path exists precisely because `raw * 1000` does not fit in 64 bits for large raws).
+- Wrapped values can be **negative**. Nothing may assume a tick count is positive.
+- **Windows is not affected** in our testing so far, which is exactly why this class of bug
+  ships: it works on every dev machine and fails on a customer's GCP instance.
+
+## The rules
+
+1. **Compare by subtraction, never directly.** Subtraction of two ticks wraps correctly in two's
+   complement; direct comparison does not.
+
+   ```csharp
+   // WRONG: fails when ticks wrap or start huge
+   if (Core.TickCount < deadline)
+
+   // RIGHT: wraparound-safe
+   if (Core.TickCount - deadline < 0)
+   ```
+
+2. **Durations are always subtractions of two readings** (`elapsed = end - start`). Never derive
+   a duration from a single absolute value.
+
+3. **No zero or sign sentinels.** `if (_lastEventAt > 0)` as "has this happened yet" breaks when
+   ticks are negative. Track "has happened" with a separate `bool` or an existing counter.
+
+4. **Seed deadline fields from a real tick, not from field initialization.** A `long _deadline;`
+   left at 0 compares wrong against a huge or negative tick. Initialize relative to the first
+   observed timestamp (see the schedule-state seeding in `Core.Setup`).
+
+5. **Store deadlines as `start + interval` only if every comparison follows rule 1.** The
+   addition may wrap; the subtraction comparison handles it.
+
+## Reviewing for it
+
+Grep the diff for `TickCount <`, `TickCount >`, `GetTimestamp() <`, and comparisons against any
+field whose name suggests a deadline (`*Until`, `*At`, `*Next*`). Each hit must be in subtraction
+form. `DateTime`/`DateTimeOffset` comparisons are unaffected; this applies only to the monotonic
+tick domain.