240118340e
## Problem The late-wake detector added in #2559 suspends idle sleeping on perfectly healthy hosts. The visible symptom is this Warning firing periodically on stable machines: > This host returned a 2ms idle wait at least 8ms late 2 time(s) in the last second; idle sleeping suspended for 5000ms Demoting it to Debug would hide the symptom but not the cost: every one of those lines means the shard dropped idle sleeping for 5s and burned a full core for no reason. The detector is what was mis-tuned. ## Cause 1 — lateness was a count, not a rate An idle loop performs **~400–500 sleeps per second** (2ms each, bounded by the 8ms wheel tick). The trip condition was `late > 1` across two consecutive one-second samples — a **0.4% tail-outlier rate**. A co-tenant burst, a page fault, or another process changing the system timer resolution clears that bar on a healthy host. A host that genuinely cannot schedule the process — throttled burstable vCPU — returns *most* of its waits late. Signal and noise were two orders of magnitude apart, and the check sat in the noise. Now gated on the proportion, with the absolute count kept as a floor: ```csharp if (late <= _lateWakeThreshold) { _consecutiveBadSamples = 0; return; } // floor if (late * 100 < sleeps * _lateWakePercent) { _consecutiveBadSamples = 0; return; } // rate ``` New `server.lateWakePercent` (default `10`). The floor is what keeps a window with only a handful of sleeps from tripping on a meaningless percentage; `server.lateWakeThreshold` keeps its existing meaning. ## Cause 2 — GC pauses were charged to the host `dev-docs/debugging-event-loop.md` already documents that the GC collects preferentially **during idle sleeps** — that is the natural pause point it looks for. So the detector was systematically measuring the GC's chosen pause point and billing it to the host's scheduler. Not an occasional coincidence; a designed-in one. ```csharp var collections = GC.CollectionCount(1); NetState.WaitForCompletion(requested); ... if (elapsed - requested >= Timer.TickRate && GC.CollectionCount(1) == collections) ``` Gen1 (which counts gen2 with it) rather than gen0 — gen0 pauses don't approach the 8ms `TickRate` bar anyway, and gating on them would discard useful samples. The second read short-circuits behind the overshoot test, so the common path costs **one** `GC.CollectionCount` per sleep: an internal counter read, single-digit nanoseconds, ~500/sec. ## Cause 3 — every backoff logged at Warning Tiered to the escalation that already existed, since a single suspension is recoverable and not something an operator can act on: | Backoff | Level | |---|---| | 1–2 | `Debug` | | 3–5 | `Warning` (now includes the sleep count and "for the Nth time running") | | ceiling | `Error`, unchanged | | recovery | `Information` (new) | Each backoff doubles the suspension, so every line is already a distinct escalation step — no further rate limiting needed. ## Drive-by The `BackoffResetAfterCleanMs` reset only ran on the path to a *new* backoff, making it unreachable for a host that recovered for good — such a host never cleared its escalation or re-armed `_loggedBackoffCeiling`. It now runs on every health sample, which is also what makes the new recovery line reachable. ## Testing Full solution builds clean, 0 warnings. No tests added: the state is private static in `Core` coupled to `_tickCount` with no injection point, and nothing covered it before — adding a seam purely to test it seemed worse than the gap. Happy to add one if reviewers disagree.
121 lines
6.6 KiB
Markdown
121 lines
6.6 KiB
Markdown
# Server Requirements
|
||
|
||
Hardware guidance for running a ModernUO shard.
|
||
|
||
## Tiers
|
||
|
||
| Use | vCPU | RAM | Storage |
|
||
|---|---|---|---|
|
||
| Development / test | 2 **dedicated** | 2 GB | SSD |
|
||
| Small live shard (< 50 concurrent) | 4 dedicated | 4 GB | NVMe |
|
||
| Medium (50–200) | 4–8 | 8 GB | NVMe |
|
||
| Large (200+) | 8+, high clock | 16 GB+ | NVMe |
|
||
|
||
These are starting points. Save size drives RAM more than player count does, and single-thread
|
||
clock speed drives tick latency more than core count does. Both are explained below.
|
||
|
||
## Dedicated vCPU, not burstable
|
||
|
||
This matters more than any other line on this page.
|
||
|
||
Budget VPS plans sold as "2 vCPU" are frequently shared or burstable: you get a CPU credit balance
|
||
or a cgroup quota, and once it is exhausted the hypervisor throttles you. Throttling shows up in
|
||
game as periodic freezes that correlate with nothing in your logs, and it is the single most common
|
||
cause of "ModernUO is laggy on my $3/month VPS".
|
||
|
||
Symptoms worth checking before blaming the server:
|
||
|
||
- Steal time above ~1% (`top`, the `%st` column on Linux)
|
||
- Lag that disappears when you move to a larger plan with the same core count
|
||
- Tick lag spikes with no matching CPU spike in the process itself
|
||
|
||
## Cores
|
||
|
||
Game logic is **single-threaded**. Every mobile, item, timer, and packet handler runs on one
|
||
thread, so a shard's headroom is bounded by how fast one core is. Two fast cores beat four slow
|
||
ones.
|
||
|
||
Cores beyond the first are used by:
|
||
|
||
- **World saves.** `world.useMultithreadedSaves` (default on) spins up `ProcessorCount - 1`
|
||
serialization workers plus one inline on the main thread. On a 2-core box that is one worker; on
|
||
a 2-core box with a large world, consider setting it to `false` so saves do not contend with the
|
||
loop.
|
||
- **The .NET runtime.** Tiered JIT compilation (heaviest in the first minutes after boot) and
|
||
background GC.
|
||
- **Everything else on the machine**, including your OS and, on Windows, antivirus.
|
||
|
||
Since ModernUO 2026 the loop sleeps when idle, so an empty shard costs roughly 1% of a core rather
|
||
than spinning. That change disproportionately helps small hosts.
|
||
|
||
## Memory
|
||
|
||
Three things dominate, and only one of them scales with players.
|
||
|
||
**World size.** A world of ~190,000 items and ~33,000 mobiles loads in about a second and is not
|
||
itself large. Items and mobiles are the cheap part.
|
||
|
||
**Saves.** Each serialization worker pre-allocates a heap sized to its share of the last save, at
|
||
roughly 1.25× total save size, and those buffers are retained afterwards. A 400 MB save therefore
|
||
implies about 500 MB of resident serialization heap on top of the live world. **This is the reason
|
||
1 GB hosts are not viable for a real shard**, even though an empty one boots fine.
|
||
|
||
**Map residency.** `TileMatrix` reads map blocks from disk on demand and caches them permanently —
|
||
there is no eviction. Memory climbs toward full-facet residency as players explore. Felucca's land
|
||
tiles alone are around 117 MB, and statics are larger.
|
||
|
||
Optional systems can add substantially more. The pathfinding prebake
|
||
(`pathfinding.prebakeMaps`) peaks above 1 GB of heap while baking. Budget for it or leave it off on
|
||
small hosts.
|
||
|
||
Network buffers are minor by comparison: 64 KB receive plus a configurable 256 KB send
|
||
(`network.sendBufferSize`) per connection, so 100 players is roughly 32 MB.
|
||
|
||
ModernUO runs **Workstation GC**, which is the right default for small hosts. Do not switch to
|
||
Server GC on a 2-core box.
|
||
|
||
## Storage
|
||
|
||
Saves are write-heavy bursts. Cheap network-attached storage with throttled IOPS will stall the
|
||
save path, and `World.WaitForWriteCompletion` blocks the loop at shutdown. Use local NVMe or SSD.
|
||
|
||
Budget disk for: the world save, plus archives and backups if `autoArchive` is enabled (retention
|
||
defaults keep 24 hourly, 30 daily, and 12 monthly copies), plus the pathfinding cache if enabled.
|
||
|
||
## Operating systems
|
||
|
||
See the README for the full supported list. Two things are worth calling out:
|
||
|
||
- **Windows Server 2012 R2 and 2016 sleep via a raised timer resolution.** Sleeping for a couple
|
||
of milliseconds prefers a high-resolution waitable timer, which requires Windows 10 1803 /
|
||
Server 2019. On older versions the ring falls back to `timeBeginPeriod(1)`, which raises the
|
||
system timer resolution to 1 ms so the plain wait timeout is accurate enough. The trade-off is a
|
||
higher interrupt rate (system-wide on those versions) — an acceptable price on a dedicated game
|
||
server, and the reason the high-resolution timer is preferred where it exists.
|
||
|
||
Only if *both* mechanisms fail does the server detect it at startup, log it, and spin instead —
|
||
the same behaviour as setting `server.eventLoopIdleWaitMs` to 0: a full core at idle, and zero
|
||
missed deadlines. A host that claims short waits but cannot deliver them is caught at runtime by
|
||
the adaptive backoff.
|
||
- **Linux kernel 6.1** or newer (Debian 12 and equivalents). io_uring is used where available, with
|
||
automatic epoll fallback.
|
||
|
||
## Tuning for a small host
|
||
|
||
| Setting | Default | Why change it |
|
||
|---|---|---|
|
||
| `server.eventLoopIdleWaitMs` | `2` | `0` never sleeps: ~98% of one core, but zero skipped timer slots and zero lag. The choice for a large shard on dedicated CPU that would rather spend a core than risk a late wake. Above `2` the wheel starts losing slots. |
|
||
| `server.lateWakeThreshold` | `1` | Floor for the backoff: idle waits the host may return a full tick late, per second, before the rate test below applies at all. Raise on a jittery host; set very high to disable the backoff. |
|
||
| `server.lateWakePercent` | `10` | Share of a second's idle waits that must come back late before idle sleeping backs off. An idle loop sleeps hundreds of times a second, so a bare count cannot tell a few tail outliers from a host that never schedules the process — a genuinely bad host misses *most* of its waits. `0` leaves `lateWakeThreshold` in sole charge. |
|
||
| `world.useMultithreadedSaves` | `true` | Set `false` on 2-core hosts so saves do not contend with the game loop. |
|
||
| `pathfinding.prebakeMaps` | varies | Leave off on memory-constrained hosts; it peaks above 1 GB while baking. |
|
||
| `network.sendBufferSize` | 256 KB | Lower it if you are memory-bound with many connections. |
|
||
| `autoArchive.*` retention | 24h/30d/12m | Reduce if disk is tight. |
|
||
|
||
## Am I undersized?
|
||
|
||
Watch the log. The server warns when the host returns idle waits late and suspends idle sleeping,
|
||
and says so at startup if the host cannot honour short waits at all. Those warnings mean the host
|
||
is not scheduling the process promptly — typical of burstable or shared vCPU plans — and no
|
||
server-side change fixes that. For anything deeper, see
|
||
[debugging-event-loop.md](debugging-event-loop.md).
|