fix(console): stop headless servers from pegging a CPU core (#2535)

## Problem

On headless Linux deployments (systemd service, Docker without a TTY, `nohup`), the ModernUO process pegs a full CPU core even when idle. It does not reproduce on Windows because that runs with an interactive console.

## Root cause

`ConsoleInputHandler` runs a background thread (named "Console Input Handler") that loops on `Console.ReadLine()`. When stdin is **not** an interactive terminal, `Console.ReadLine()` returns `null` at end-of-stream **immediately** on every call, so the loop `continue`s in a tight spin — one core at 100%.

Reproduced in a container running the actual distribution: the "Console Input Handler" thread sat at ~90% CPU on a headless boot; with a blocking stdin it dropped to idle.

## Fix

1. **Detect headless once at startup:** `Core.Headless = Console.IsInputRedirected`.
2. **Extract a testable `ConsoleInputPump`** that owns the input stream: per line read, it *atomically* (under one lock) either delivers the line to a waiting prompt or dispatches a console command, and it **ends on EOF instead of spinning**. Cleanup runs unconditionally in a `finally`, so a pending prompt is always released (never hangs). Replaces the old `async void` loop and the fragile `_expectUserInput` / two-`AutoResetEvent` / `_input` handshake.
3. **`ConsoleInputHandler` becomes a thin headless-aware facade** over the pump. Headless: the reader thread never starts (`Console input disabled (headless: stdin is not a TTY).`), and `ReadLine()` throws a fatal `HeadlessConsoleInputException`.
4. **Data-gating and first-boot prompts** (deserialization "delete bad types? y/n", save-conflict, config/expansion setup) now route through `ConsoleInputHandler.ReadLine()`, so a headless server crashes fatal with a clear message instead of reading `null` (previously an NRE or a silent wrong branch).

Design decision (model b): headless servers are expected to be supplied with configuration/save data (including the owner account); interactive prompts when headless are fatal by design.

## Testing

- New `ConsoleInputPumpTests` (5 tests): EOF ends the loop without spinning; command dispatch; a pending prompt receives the next line; EOF while a prompt is pending completes it with `null` (no hang); a throwing command lookup does not hang a pending prompt. The tests synchronize on real pump state (no `Thread.Sleep`), so they are deterministic on slow CI.
- Full `Server.Tests`: no new failures introduced.

## End-to-end verification (Docker, real distribution)

| | Console Input Handler thread | Container CPU |
|---|---|---|
| Before fix (headless boot) | ~90% | ~199% (2 cores) |
| After fix (headless boot) | **not started** | **~11%** |

After the fix, a headless boot logs `Console input disabled (headless: stdin is not a TTY).`, loads the world normally, and idles instead of spinning.
This commit is contained in:
Kamron Batman 2026-07-16 18:52:43 -07:00 committed by GitHub
parent f4a771c19d
commit 7434ed7ee1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 354 additions and 74 deletions

View file

@ -0,0 +1,139 @@
using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Server;
using Xunit;
namespace Server.Tests;
// These tests exercise the pump's cross-thread rendezvous directly, so they block on
// bounded-timeout Task.Wait() calls by design rather than using async/await.
#pragma warning disable xUnit1031
public class ConsoleInputPumpTests
{
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(5);
// Deterministic synchronization: spin until a real condition holds (the pump's
// rendezvous state or the reader's progress) instead of sleeping a fixed interval,
// which races on slow/loaded CI runners.
private static void WaitUntil(Func<bool> condition, string message) =>
Assert.True(SpinWait.SpinUntil(condition, Timeout), message);
// A TextReader whose ReadLine() blocks until a line is fed, and returns null after Complete().
private sealed class BlockingTextReader : TextReader
{
private readonly BlockingCollection<string> _lines = new();
private int _readCalls;
// Number of times ReadLine() has been entered — lets a test wait until the pump's
// reader loop is blocked waiting for the next line (e.g. after consuming a command).
public int ReadCallCount => Volatile.Read(ref _readCalls);
public void Feed(string line) => _lines.Add(line);
public void Complete() => _lines.CompleteAdding();
public override string ReadLine()
{
Interlocked.Increment(ref _readCalls);
try
{
return _lines.Take();
}
catch (InvalidOperationException)
{
return null; // CompleteAdding + empty => EOF
}
}
}
[Fact]
public void Eof_ends_loop_without_spinning()
{
var pump = new ConsoleInputPump(new StringReader(""), _ => null);
var ran = Task.Run(pump.Run);
Assert.True(ran.Wait(Timeout), "Run() did not return on EOF");
Assert.False(pump.Running);
}
[Fact]
public void Recognized_line_dispatches_command()
{
var invokedWith = new TaskCompletionSource<string>();
Action<string> onPing = arg => invokedWith.TrySetResult(arg);
var pump = new ConsoleInputPump(
new StringReader("ping hello\n"),
cmd => cmd == "ping" ? onPing : null
);
Task.Run(pump.Run);
Assert.True(invokedWith.Task.Wait(Timeout), "command not dispatched");
Assert.Equal("hello", invokedWith.Task.Result);
}
[Fact]
public void Pending_prompt_receives_next_line()
{
var reader = new BlockingTextReader();
var pump = new ConsoleInputPump(reader, _ => null);
Task.Run(pump.Run);
// Register the prompt, then wait until it is actually pending before feeding a
// line — otherwise the line could be consumed as a command before registration.
var prompt = Task.Run(pump.ReadLine);
WaitUntil(() => pump.HasPendingPrompt, "prompt did not register");
reader.Feed("the answer");
Assert.True(prompt.Wait(Timeout), "prompt did not receive a line");
Assert.Equal("the answer", prompt.Result);
reader.Complete();
}
[Fact]
public void Eof_while_prompt_pending_completes_prompt_with_null()
{
var reader = new BlockingTextReader();
var pump = new ConsoleInputPump(reader, _ => null);
Task.Run(pump.Run);
var prompt = Task.Run(pump.ReadLine);
WaitUntil(() => pump.HasPendingPrompt, "prompt did not register");
reader.Complete(); // EOF while prompt is waiting
Assert.True(prompt.Wait(Timeout), "prompt hung on EOF");
Assert.Null(prompt.Result);
Assert.False(pump.Running);
}
[Fact]
public void Throwing_lookup_does_not_hang_pending_prompt()
{
var reader = new BlockingTextReader();
var pump = new ConsoleInputPump(reader, _ => throw new InvalidOperationException("boom"));
Task.Run(pump.Run);
// Wait until the reader loop is blocked on its first read, then feed a bad command.
WaitUntil(() => reader.ReadCallCount >= 1, "reader did not start");
reader.Feed("badcommand");
// Wait until the loop has consumed that command (throwing lookup swallowed) and is
// blocked waiting for the NEXT line — a second ReadLine() entry. This guarantees
// the command was dispatched (not delivered to a prompt) before we register one.
WaitUntil(() => reader.ReadCallCount >= 2, "bad command was not consumed");
var prompt = Task.Run(pump.ReadLine);
WaitUntil(() => pump.HasPendingPrompt, "prompt did not register");
reader.Complete(); // EOF while prompt is waiting
Assert.True(prompt.Wait(Timeout), "prompt hung after throwing lookup");
Assert.Null(prompt.Result);
Assert.False(pump.Running);
}
}
#pragma warning restore xUnit1031

View file

@ -23,7 +23,7 @@ public static class ExpansionConfigurationPrompts
do
{
Console.Write("[enter for {0}]> ", maxExpansionName);
var input = Console.ReadLine();
var input = ConsoleInputHandler.ReadLine();
Expansion expansion;
if (string.IsNullOrWhiteSpace(input))
@ -77,7 +77,7 @@ public static class ExpansionConfigurationPrompts
do
{
OutputSelectedMaps(expansion, selectedMaps);
lastInput = Console.ReadLine()?.TrimEnd();
lastInput = ConsoleInputHandler.ReadLine()?.TrimEnd();
if (string.IsNullOrWhiteSpace(lastInput))
{

View file

@ -16,7 +16,7 @@ public static class ServerConfigurationPrompts
do
{
Console.Write("{0}> ", directories.Count > 0 ? "[enter to finish]" : " ");
var directory = Console.ReadLine();
var directory = ConsoleInputHandler.ReadLine();
if (string.IsNullOrWhiteSpace(directory))
{
break;
@ -55,7 +55,7 @@ public static class ServerConfigurationPrompts
{
// IP:Port?
Console.Write("[{0}]> ", ips.Count > 0 ? "enter to finish" : "0.0.0.0:2593");
var ipStr = Console.ReadLine();
var ipStr = ConsoleInputHandler.ReadLine();
IPEndPoint ip;
if (string.IsNullOrWhiteSpace(ipStr))
@ -102,7 +102,7 @@ public static class ServerConfigurationPrompts
do
{
Console.Write("[ModernUO]> ");
serverName = Console.ReadLine();
serverName = ConsoleInputHandler.ReadLine();
if (string.IsNullOrWhiteSpace(serverName))
{

View file

@ -24,13 +24,9 @@ namespace Server;
public static class ConsoleInputHandler
{
private static readonly AutoResetEvent _receivedUserInput = new(false);
private static readonly AutoResetEvent _endUserInput = new(false);
private static bool _initialized;
private static bool _expectUserInput;
private static ConsoleInputPump _pump;
private static readonly Dictionary<string, ConsoleCommand> _inputCommands = new();
private static string[] _commandDescriptions;
private static string _input;
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static void RegisterCommand(string command, string description, Action<string> function) =>
@ -73,7 +69,6 @@ public static class ConsoleInputHandler
lock (_inputCommands)
{
var action = _inputCommands.GetValueOrDefault(command)?.Function;
_commandDescriptions = null;
return action;
}
}
@ -86,9 +81,15 @@ public static class ConsoleInputHandler
[CallPriority(0)]
public static void Initialize()
{
_initialized = true;
if (Core.Headless)
{
logger.Information("Console input disabled (headless: stdin is not a TTY).");
return;
}
new Thread(ProcessConsoleInput)
_pump = new ConsoleInputPump(Console.In, GetInputCommand, logger);
new Thread(_pump.Run)
{
IsBackground = true,
Name = "Console Input Handler"
@ -150,69 +151,21 @@ public static class ConsoleInputHandler
private static readonly ILogger logger = LogFactory.GetLogger(typeof(ConsoleInputHandler));
private static async void ProcessConsoleInput()
{
while (!Core.Closing)
{
string input;
try
{
input = Console.ReadLine()?.Trim();
}
catch
{
logger.Warning("Console commands have been disabled due to an error.");
_initialized = false;
return;
}
if (Volatile.Read(ref _expectUserInput))
{
_input = input;
_receivedUserInput.Set();
_endUserInput.WaitOne();
continue;
}
if (string.IsNullOrEmpty(input))
{
continue;
}
var splitInput = input.Split(' ', 2);
var command = splitInput[0].ToLower();
try
{
Action<string> action;
lock (_inputCommands)
{
action = _inputCommands.GetValueOrDefault(command)?.Function;
}
action?.Invoke(splitInput.Length > 1 ? splitInput[1] : string.Empty);
}
catch (Exception e)
{
logger.Error(e, "Failed to execute console command: {Command}", input);
}
}
}
public static string ReadLine()
{
if (!_initialized)
if (Core.Headless)
{
throw new HeadlessConsoleInputException("ConsoleInputHandler.ReadLine");
}
// Early startup (before Initialize) or after the loop ended: read directly.
var pump = _pump;
if (pump is not { Running: true })
{
return Console.ReadLine();
}
Volatile.Write(ref _expectUserInput, true);
_receivedUserInput.WaitOne();
var line = _input;
Volatile.Write(ref _expectUserInput, false);
_endUserInput.Set();
return line;
return pump.ReadLine();
}
private class ConsoleCommand(string[] commands, string description, Action<string> function)

View file

@ -0,0 +1,162 @@
using System;
using System.IO;
using System.Threading;
using Server.Logging;
namespace Server;
/// <summary>
/// Owns a console input stream. Each line read is atomically either delivered to a
/// waiting <see cref="ReadLine"/> caller (a prompt) or dispatched as a command via the
/// supplied lookup. EOF ends the loop instead of spinning. Correct-by-construction:
/// the prompt-vs-command decision is made under a single lock at the moment a line is read.
/// Command handlers dispatched by <see cref="Run"/> execute on the reader thread and
/// must never call <see cref="ReadLine"/> — doing so would deadlock the pump (the reader
/// thread would be blocked waiting on itself to read the next line).
/// </summary>
internal sealed class ConsoleInputPump
{
private readonly TextReader _input;
private readonly Func<string, Action<string>> _lookup;
private readonly Server.Logging.ILogger _logger;
private readonly Lock _gate = new();
private readonly AutoResetEvent _promptDelivered = new(false);
private bool _promptPending;
private string _promptResult;
private volatile bool _running = true;
public ConsoleInputPump(TextReader input, Func<string, Action<string>> lookup, Server.Logging.ILogger logger = null)
{
_input = input ?? throw new ArgumentNullException(nameof(input));
_lookup = lookup ?? throw new ArgumentNullException(nameof(lookup));
_logger = logger;
}
public bool Running => _running;
// Test-observable: true while a ReadLine() caller is registered and waiting for the
// next line. Lets tests synchronize on the rendezvous state instead of sleeping.
internal bool HasPendingPrompt
{
get
{
lock (_gate)
{
return _promptPending;
}
}
}
public void Run()
{
try
{
while (_running && !Core.Closing)
{
string line;
try
{
line = _input.ReadLine();
}
catch
{
_logger?.Warning("Console commands have been disabled due to an error.");
break;
}
if (line == null)
{
break; // EOF — never spin
}
bool isCommand;
lock (_gate)
{
if (_promptPending)
{
_promptResult = line;
_promptPending = false;
_promptDelivered.Set();
isCommand = false;
}
else
{
isCommand = true;
}
}
if (!isCommand)
{
continue;
}
var trimmed = line.Trim();
if (trimmed.Length == 0)
{
continue;
}
var split = trimmed.Split(' ', 2);
try
{
var action = _lookup(split[0].ToLower());
action?.Invoke(split.Length > 1 ? split[1] : string.Empty);
}
catch (Exception e)
{
_logger?.Error(e, "Failed to execute console command: {Command}", line);
}
}
}
finally
{
_running = false;
ReleasePendingPrompt(null);
}
}
/// <summary>
/// Blocks the calling thread until the next console line is available, or until the
/// pump stops (returning <c>null</c>). Intended for a single, sequential caller at a
/// time — ModernUO's console prompts run one after another during startup/steps.
/// Concurrent callers are not supported: a second caller overlapping with a pending
/// prompt will race with it for the next line. Must not be called from the reader
/// thread (i.e. from within a command handler dispatched by <see cref="Run"/>), as
/// that would deadlock the pump.
/// </summary>
public string ReadLine()
{
lock (_gate)
{
if (!_running)
{
return null;
}
_promptResult = null;
_promptPending = true;
}
_promptDelivered.WaitOne();
lock (_gate)
{
return _promptResult;
}
}
private void ReleasePendingPrompt(string result)
{
lock (_gate)
{
if (_promptPending)
{
_promptResult = result;
_promptPending = false;
_promptDelivered.Set();
}
}
}
}

View file

@ -0,0 +1,18 @@
using System;
namespace Server;
/// <summary>
/// Thrown when interactive console input is required but the server is running
/// headless (stdin is not a TTY). Rides the fatal-shutdown path.
/// </summary>
public sealed class HeadlessConsoleInputException : Exception
{
public HeadlessConsoleInputException(string context)
: base(
$"Interactive console input required but the server is headless (stdin is not a TTY). " +
$"Pre-supply the required configuration/save data. Prompt: {context}."
)
{
}
}

View file

@ -147,6 +147,8 @@ public static class Core
public static bool Closing => ClosingTokenSource.IsCancellationRequested;
public static bool Headless { get; private set; }
public static int GlobalUpdateRange { get; set; } = 18;
public static int GlobalMaxUpdateRange { get; set; } = 24;
@ -258,7 +260,7 @@ public static class Core
// ignored
}
if (!close)
if (!close && !Core.Headless)
{
Console.WriteLine("This exception is fatal, press return to exit");
ConsoleInputHandler.ReadLine();
@ -407,6 +409,12 @@ public static class Core
Console.CancelKeyPress += Console_CancelKeyPressed;
Headless = Console.IsInputRedirected;
if (Headless)
{
logger.Information("Headless mode detected (stdin is not a TTY); interactive console input is disabled.");
}
// LibDeflate is not thread safe, so we need to create a new instance for each thread
var standard = Deflate.Standard;
AppDomain.CurrentDomain.ProcessExit += (_, _) => standard.Dispose();

View file

@ -166,7 +166,7 @@ public class GenericEntityPersistence<T> : GenericPersistence, IGenericEntityPer
Console.Write($"Error: Type '{typeName}' was {issue}. Delete all of those types? (y/n): ");
if (Console.ReadLine().InsensitiveEquals("y"))
if (ConsoleInputHandler.ReadLine().InsensitiveEquals("y"))
{
Console.WriteLine("Loading...");
return null;
@ -430,7 +430,7 @@ public class GenericEntityPersistence<T> : GenericPersistence, IGenericEntityPer
if (!deleteAllFailures)
{
Console.Write("Delete the object and continue? (y/n/a): ");
var pressedKey = Console.ReadLine();
var pressedKey = ConsoleInputHandler.ReadLine();
if (pressedKey.InsensitiveEquals("a"))
{

View file

@ -104,7 +104,7 @@ public abstract class GenericPersistence : Persistence, IGenericSerializable
Console.WriteLine(error);
Console.Write("Skip this file and continue? (y/n): ");
var y = Console.ReadLine();
var y = ConsoleInputHandler.ReadLine();
if (!y.InsensitiveEquals("y"))
{