fix: Harden Advanced Search: crash-safety, autosave, correct results & worker fixes (#2543)

## Summary

Hardens the **Advanced Search** engine (`Projects/UOContent/Engines/Advanced Search/`) — the GM entity finder that fans searches across background worker threads. A code review surfaced 14 defects (A–N), including a shard-crasher reachable from a single admin typo and a path that silently disables autosave for the rest of the shard's uptime. Each behavioral fix ships with a test.

Full `UOContent.Tests` suite: **530/530 green** (21 new AdvancedSearch tests).

## Fixes

### Crash / data-loss
- **A — Shard crash on a malformed Property Test.** `AdvancedSearchThreadWorker.Execute` had no `try/catch` and the worker `Thread` is foreground, so a parse throw (`Hits>abc`, `Layer=onehanded` — `Enum.Parse` was case-sensitive, `Hits>1@` — empty sub-expression indexing) terminated the process. Now: `ParseValue`/`CompareValues` use `TryParse`/`Enum.TryParse(ignoreCase)` and return no-match instead of throwing; the per-entity filter is wrapped in `try/catch` (logs + skips); empty expressions are guarded.
- **C — Overlapping searches corrupt state + brick autosave.** `_threadWorkers`/`_threadId` were `static` but `DoSearch` is an instance method; a second search (double-click / two admins) stomped shared worker state and could leave a drain waiting forever on the shared `AutoResetEvent`, so `AutoSave.SavesEnabled` was never restored. Now: an `Interlocked` re-entrancy guard rejects concurrent searches.
- **G — Autosave restore not guaranteed.** The restore lived only in the success callback. Now it's in a `finally` (plus an outer `catch` covering the synchronous setup and a `catch` on the drain body), so autosave + the guard are always released.

### Wrong results
- **D — `@`/`|` operator precedence.** `a@b|c` evaluated as `a && (b || c)` instead of `(a && b) || c`. OR now binds looser than AND (`AdvancedSearchUtilities.EvaluateBoolean`, unit-tested).
- **E — Descending sort, partial last page rendered blank** (the index decreased in descending mode and the `break` early-out killed the loop). Now a bounded `VisibleCount`-driven loop renders the last page in both directions.
- **F — Deleted entities** were not skipped (ghost rows). Now `DoEntitySearch` skips `entity.Deleted`.
- **N — Reference-type comparisons** threw (`Comparer<T>.Default.Compare` on non-`IComparable`) and compared references to a string. Now equality is by value and ordering is guarded to `IComparable` (no throw).

### Worker perf / hardening
- **H** busy-spin → `Thread.Yield()` in the drain; **I** `GetProperties()` cached per `Type`; **J** `HandleValidInternal` moved behind the cheap map/range/region filters; **K** worker threads are `IsBackground` + `Exit()` tolerates an already-terminated worker; **L** `_filter == null` guard; **M** consistent `Volatile` access on `_pause`/`_exit`.

### Documented
- **B** — the residual worker/event-loop read race is documented on `AdvancedSearchThreadWorker`: workers read live entity state concurrently with the loop, so value-type reads may be stale-but-safe and getter exceptions are swallowed; fully eliminating it would require snapshotting entity fields on the main thread (deferred).

## Notes
- New test-only seams (`TryBeginSearch`/`EndSearch`/`IsSearchInProgress`/`VisibleCount`/`TryParseValue`/`EvaluateBoolean`) are `internal` via the existing `InternalsVisibleTo("UOContent.Tests")`.
- Dead `public ParseValue<T>` removed.
- `ConcurrentDictionary` for the reflection cache is intentional — these workers are genuinely parallel.
This commit is contained in:
Kamron Batman 2026-07-21 07:51:06 -07:00 committed by GitHub
parent 858c1d18bc
commit 1e97ed50f6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 842 additions and 209 deletions

View file

@ -3,13 +3,26 @@ using System.Collections.Concurrent;
using System.Reflection;
using System.Threading;
using Server.Items;
using Server.Logging;
using Server.Mobiles;
using Server.Multis;
namespace Server.Engines.AdvancedSearch;
/// <summary>
/// Filters entities on a background thread while the main loop keeps mutating them — an
/// intentional, bounded race. Reads of live <see cref="Item"/>/<see cref="Mobile"/> state are
/// unsynchronized, so a torn <see cref="Point3D"/> read may report stale coordinates, and any
/// getter that throws mid-read is caught per-entity in <see cref="DoEntitySearch"/> and skipped.
/// Results are best-effort and may omit a concurrently modified entity, but never fault or corrupt
/// server state. Eliminating the race would require snapshotting each read field onto the main
/// thread before handing entities off; that is deferred.
/// </summary>
public class AdvancedSearchThreadWorker
{
private static readonly ILogger _logger = LogFactory.GetLogger(typeof(AdvancedSearchThreadWorker));
private static readonly ConcurrentDictionary<Type, PropertyInfo[]> _propCache = new();
private readonly Thread _thread;
private readonly AutoResetEvent _startEvent; // Main thread tells the thread to start working
private readonly AutoResetEvent _stopEvent; // Main thread waits for the worker finish draining
@ -26,7 +39,10 @@ public class AdvancedSearchThreadWorker
_startEvent = new AutoResetEvent(false);
_stopEvent = new AutoResetEvent(false);
_entities = new ConcurrentQueue<IEntity>();
_thread = new Thread(Execute);
_thread = new Thread(Execute)
{
IsBackground = true
};
_thread.Start(this);
}
@ -52,10 +68,16 @@ public class AdvancedSearchThreadWorker
public void Exit()
{
_exit = true;
Volatile.Write(ref _exit, true);
Wake(WorldLocation.Zero, null, null, null);
Sleep();
// Tolerate a worker that has already terminated (e.g. Core.Closing raced us) so
// shutdown can't deadlock waiting on a stopEvent that will never be set.
if (_thread.IsAlive)
{
Sleep();
}
}
public void Push(IEntity entity)
@ -88,12 +110,17 @@ public class AdvancedSearchThreadWorker
worker._filter = null;
break;
}
else
{
// Transiently empty but not yet paused: yield rather than busy-spin.
Thread.Yield();
}
}
worker._stopEvent.Set(); // Allow the main thread to continue now that we are finished
worker._pause = false;
Volatile.Write(ref worker._pause, false);
if (Core.Closing || worker._exit)
if (Core.Closing || Volatile.Read(ref worker._exit))
{
return;
}
@ -102,9 +129,28 @@ public class AdvancedSearchThreadWorker
private AdvancedSearchResult DoEntitySearch(IEntity entity)
{
if (_filter.HideValidInternalMap)
if (entity == null || entity.Deleted)
{
HandleValidInternal(entity);
return null;
}
try
{
return DoEntitySearchCore(entity);
}
catch (Exception ex)
{
_logger.Warning(ex, "AdvancedSearch: filter threw for {Entity}; skipping", entity);
return null;
}
}
private AdvancedSearchResult DoEntitySearchCore(IEntity entity)
{
if (_filter == null)
{
// Exit() clears the filter; a straggler entity dequeued after teardown bails here.
return null;
}
// Check for valid map
@ -138,6 +184,12 @@ public class AdvancedSearchThreadWorker
return null;
}
// After the cheap filters, so non-qualifying entities skip the house/keyring enumeration.
if (_filter.HideValidInternalMap)
{
HandleValidInternal(entity);
}
if (entity is Mobile mobile)
{
return DoMobileSearch(mobile);
@ -296,27 +348,17 @@ public class AdvancedSearchThreadWorker
}
}
private static bool EvaluateRecursive(IEntity entity, ReadOnlySpan<char> span)
{
var atIndex = span.IndexOf('@');
var orIndex = span.IndexOf('|');
if (atIndex == -1 && orIndex == -1)
{
return EvaluateSingleExpression(entity, span);
}
var result = atIndex != -1;
var splitIndex = result ? atIndex : orIndex;
var left = EvaluateRecursive(entity, span.Slice(0, splitIndex));
var right = EvaluateRecursive(entity, span.Slice(splitIndex + 1));
return result ? left && right : left || right;
}
private static bool EvaluateRecursive(IEntity entity, ReadOnlySpan<char> span) =>
AdvancedSearchUtilities.EvaluateBoolean(span, entity, static (e, leaf) => EvaluateSingleExpression(e, leaf));
private static bool EvaluateSingleExpression(IEntity entity, ReadOnlySpan<char> expression)
{
expression = expression.Trim();
if (expression.Length == 0)
{
return false;
}
var negate = false;
if (expression[0] == '~')
{
@ -338,7 +380,7 @@ public class AdvancedSearchThreadWorker
return false;
}
var properties = entity.GetType().GetProperties();
var properties = _propCache.GetOrAdd(entity.GetType(), static t => t.GetProperties());
PropertyInfo property = null;
for (var i = 0; i < properties.Length; ++i)
{