ModernUO/Projects/UOContent/Engines/Advanced Search/AdvancedSearchThreadWorker.cs
Kamron Batman 1a9cec1dbb
fix(advancedsearch): clear pause and sample exit before signaling the drain (#2549)
## Summary

`AdvancedSearchThreadWorker.Execute` signals `_stopEvent` **before** clearing `_pause` and **before** reading the exit condition. `Sleep()` unblocks the instant that signal fires, so the owning thread can begin the next cycle while the worker is still finishing the previous one — and the worker's two trailing operations then land on the new cycle's state.

`SerializationThreadWorker` already orders the same handshake correctly and documents why (`Projects/Server/Serialization/SerializationThreadWorker.cs`):

```csharp
// The owning thread may start another pause cycle the moment _stopEvent is set
// (Exit does exactly that). Clear _pause and sample the exit condition before
// signaling, or the new cycle's pause request is clobbered / its Sleep orphaned.
var exiting = Core.Closing || worker._exit;
Volatile.Write(ref worker._pause, false);
worker._stopEvent.Set();
```

This applies the same ordering to the search worker. Three lines; no behavior change on the happy path.

## The two failures

**Reuse hang.** The next cycle's `Wake`/`Push`/`Sleep` writes `_pause = true`, then the worker's stale `_pause = false` lands on top of it. The inner loop never observes `pauseRequested`, its queue is already empty, and it spins on `Thread.Yield()` forever — so the owning thread's next `Sleep()` waits on a `_stopEvent` that is never set again. A single search wakes each worker exactly once, so this only surfaces once `_threadWorkers` is reused by a later search.

**Orphaned `Exit()`.** `Exit()` sets `_exit`, `Wake()`s, then `Sleep()`s — the moment the drain's `Sleep()` returns. Reading `_exit` *after* the signal, the worker can observe that fresh `_exit`, return without ever consuming the `Wake`, and leave `Exit()`'s `Sleep()` waiting on a signal nobody will send. The `_thread.IsAlive` guard doesn't close this: the thread passes the check and returns immediately after.

## Verification

Verified with two throwaway timing tests — 25k reuse cycles and 2k drain-then-`Exit` cycles, each under a bounded wait:

| ordering | result |
|---|---|
| previous | `Failed: 2, Passed: 3` — both reproduce, cleanly at the 20s bound |
| this PR | 3 consecutive runs, 5/5, ~0.6s |

**Those tests are deliberately not included.** Their reproduction threshold is a property of one machine's scheduler — at 2k and 200 cycles the buggy build passed — so as permanent tests they'd cost ~560ms and 2000 thread creations on every suite run for a guarantee that may not hold on a CI runner. The ordering is protected the same way `SerializationThreadWorker`'s is: by the comment at the call site.

`UOContent.Tests`: **597/597**.
2026-07-25 15:32:06 -07:00

410 lines
12 KiB
C#

using System;
using System.Collections.Concurrent;
using System.Reflection;
using System.Threading;
using Server.Items;
using Server.Logging;
using Server.Mobiles;
using Server.Multis;
namespace Server.Engines.AdvancedSearch;
/// <summary>
/// Filters entities on a background thread while the main loop keeps mutating them — an
/// intentional, bounded race. Reads of live <see cref="Item"/>/<see cref="Mobile"/> state are
/// unsynchronized, so a torn <see cref="Point3D"/> read may report stale coordinates, and any
/// getter that throws mid-read is caught per-entity in <see cref="DoEntitySearch"/> and skipped.
/// Results are best-effort and may omit a concurrently modified entity, but never fault or corrupt
/// server state. Eliminating the race would require snapshotting each read field onto the main
/// thread before handing entities off; that is deferred.
/// </summary>
public class AdvancedSearchThreadWorker
{
private static readonly ILogger _logger = LogFactory.GetLogger(typeof(AdvancedSearchThreadWorker));
private static readonly ConcurrentDictionary<Type, PropertyInfo[]> _propCache = new();
private readonly Thread _thread;
private readonly AutoResetEvent _startEvent; // Main thread tells the thread to start working
private readonly AutoResetEvent _stopEvent; // Main thread waits for the worker finish draining
private bool _pause;
private bool _exit;
private readonly ConcurrentQueue<IEntity> _entities;
private ConcurrentQueue<AdvancedSearchResult> _results;
private ConcurrentQueue<IEntity> _ignoreQueue;
private WorldLocation _worldLocation;
private AdvancedSearchFilter _filter;
public AdvancedSearchThreadWorker()
{
_startEvent = new AutoResetEvent(false);
_stopEvent = new AutoResetEvent(false);
_entities = new ConcurrentQueue<IEntity>();
_thread = new Thread(Execute)
{
IsBackground = true
};
_thread.Start(this);
}
public void Wake(
WorldLocation worldLocation,
AdvancedSearchFilter filter,
ConcurrentQueue<AdvancedSearchResult> results,
ConcurrentQueue<IEntity> ignoreQueue
)
{
_worldLocation = worldLocation;
_filter = filter;
_ignoreQueue = ignoreQueue;
_results = results;
_startEvent.Set();
}
public void Sleep()
{
Volatile.Write(ref _pause, true);
_stopEvent.WaitOne();
}
public void Exit()
{
Volatile.Write(ref _exit, true);
Wake(WorldLocation.Zero, null, null, null);
// Tolerate a worker that has already terminated (e.g. Core.Closing raced us) so
// shutdown can't deadlock waiting on a stopEvent that will never be set.
if (_thread.IsAlive)
{
Sleep();
}
}
public void Push(IEntity entity)
{
_entities.Enqueue(entity);
}
private static void Execute(object obj)
{
var worker = (AdvancedSearchThreadWorker)obj;
var reader = worker._entities;
while (worker._startEvent.WaitOne())
{
while (true)
{
var pauseRequested = Volatile.Read(ref worker._pause);
if (reader.TryDequeue(out var entity))
{
var result = worker.DoEntitySearch(entity);
if (result != null)
{
worker._results?.Enqueue(result);
}
}
else if (pauseRequested) // Break when finished
{
worker._results = null;
worker._filter = null;
break;
}
else
{
// Transiently empty but not yet paused: yield rather than busy-spin.
Thread.Yield();
}
}
// The owning thread may start another cycle the moment _stopEvent is set (Exit does exactly
// that). Clear _pause and sample the exit condition before signaling, or the new cycle's
// pause request is clobbered / its Sleep orphaned. Matches SerializationThreadWorker.
var exiting = Core.Closing || Volatile.Read(ref worker._exit);
Volatile.Write(ref worker._pause, false);
worker._stopEvent.Set(); // Allow the main thread to continue now that we are finished
if (exiting)
{
return;
}
}
}
private AdvancedSearchResult DoEntitySearch(IEntity entity)
{
if (entity == null || entity.Deleted)
{
return null;
}
try
{
return DoEntitySearchCore(entity);
}
catch (Exception ex)
{
_logger.Warning(ex, "AdvancedSearch: filter threw for {Entity}; skipping", entity);
return null;
}
}
private AdvancedSearchResult DoEntitySearchCore(IEntity entity)
{
if (_filter == null)
{
// Exit() clears the filter; a straggler entity dequeued after teardown bails here.
return null;
}
// Check for valid map
if (_filter.FilterFelucca && entity.Map != Map.Felucca ||
_filter.FilterTrammel && entity.Map != Map.Trammel ||
_filter.FilterIlshenar && entity.Map != Map.Ilshenar ||
_filter.FilterMalas && entity.Map != Map.Malas ||
_filter.FilterTokuno && entity.Map != Map.Tokuno ||
_filter.FilterTerMur && entity.Map != Map.TerMur ||
_filter.FilterInternalMap && entity.Map != Map.Internal ||
_filter.FilterNullMap && entity.Map != null)
{
return null;
}
var location = (entity as Item)?.GetWorldLocation() ?? entity.Location;
if (_filter.FilterRange &&
(_filter.Range == null ||
_filter.Range < 0 ||
entity.Map != _worldLocation.Map ||
!Utility.InRange(_worldLocation.Location, location, _filter.Range.Value)))
{
return null;
}
if (_filter.FilterRegion &&
(string.IsNullOrWhiteSpace(_filter.RegionName) ||
!Region.Find(location, entity.Map).IsPartOf(_filter.RegionName)))
{
return null;
}
// After the cheap filters, so non-qualifying entities skip the house/keyring enumeration.
if (_filter.HideValidInternalMap)
{
HandleValidInternal(entity);
}
if (entity is Mobile mobile)
{
return DoMobileSearch(mobile);
}
if (entity is Item item)
{
return DoItemSearch(item);
}
return null;
}
private static bool IsValidInternal(Item item)
{
if (item.Parent != null || item.HeldBy != null)
{
return true;
}
if (item is Fists
or MountItem
or EffectItem
or MovingCrate
or BaseDockedBoat
or BaseBoat
or Plank
or TillerMan
or Hold)
{
return true;
}
// DisplayCache container
if (item.GetType().DeclaringType == typeof(GenericBuyInfo))
{
return true;
}
return false;
}
private AdvancedSearchResult DoItemSearch(Item item)
{
if (_filter.FilterName && !string.IsNullOrWhiteSpace(_filter.Name) && !(item.Name ?? item.ItemData.Name).InsensitiveContains(_filter.Name))
{
return null;
}
if (_filter.HideValidInternalMap && item.Map == Map.Internal && !IsValidInternal(item))
{
return null;
}
if (_filter.FilterPropertyTest &&
(string.IsNullOrWhiteSpace(_filter.PropertyTest) || !EvaluateRecursive(item, _filter.PropertyTest)))
{
return null;
}
return new AdvancedSearchResult(item.Name ?? item.ItemData.Name, item.GetType(), item.Location, item.Map, item.RootParent)
{
Entity = item,
};
}
private AdvancedSearchResult DoMobileSearch(Mobile mobile)
{
if (_filter.FilterName && !string.IsNullOrWhiteSpace(_filter.Name) && !mobile.Name.InsensitiveContains(_filter.Name))
{
return null;
}
if (_filter.HideValidInternalMap && mobile.Map == Map.Internal && !IsValidInternal(mobile))
{
return null;
}
if (_filter.FilterPropertyTest &&
(string.IsNullOrWhiteSpace(_filter.PropertyTest) || !EvaluateRecursive(mobile, _filter.PropertyTest)))
{
return null;
}
return new AdvancedSearchResult(mobile.Name, mobile.GetType(), mobile.Location, mobile.Map, null)
{
Entity = mobile
};
}
private static bool IsValidInternal(Mobile m)
{
// Logged out players
if (m.Account != null)
{
return true;
}
// Stabled pets
if (m is BaseCreature creature && creature.IsStabled)
{
return true;
}
// Internalized vendors
if (m is PlayerVendor playerVendor && playerVendor.House != null)
{
return true;
}
// Currently mounted creatures
if (m is IMount mount && mount.Rider != null)
{
return true;
}
return false;
}
public void HandleValidInternal(IEntity entity)
{
if (entity is CommodityDeed deed && deed.Commodity != null && deed.Commodity.Map == Map.Internal)
{
_ignoreQueue.Enqueue(entity);
return;
}
// Keys don't have a backreference, so we just ignore them for now
if (entity is KeyRing keyring && keyring.Keys?.Count > 0)
{
foreach (var k in keyring.Keys)
{
_ignoreQueue.Enqueue(k);
}
return;
}
if (entity is BaseHouse house)
{
foreach (var relEntity in house.RelocatedEntities)
{
if (relEntity.Entity is Item)
{
_ignoreQueue.Enqueue(relEntity.Entity);
}
}
foreach (var inventory in house.VendorInventories)
{
foreach (var subItem in inventory.Items)
{
_ignoreQueue.Enqueue(subItem);
}
}
}
}
private static bool EvaluateRecursive(IEntity entity, ReadOnlySpan<char> span) =>
AdvancedSearchUtilities.EvaluateBoolean(span, entity, static (e, leaf) => EvaluateSingleExpression(e, leaf));
private static bool EvaluateSingleExpression(IEntity entity, ReadOnlySpan<char> expression)
{
expression = expression.Trim();
if (expression.Length == 0)
{
return false;
}
var negate = false;
if (expression[0] == '~')
{
negate = true;
expression = expression[1..];
}
var operatorSpan = AdvancedSearchUtilities.FindOperatorIndex(expression, out var operatorIndex);
if (operatorSpan.Length == 0)
{
return false;
}
var propertyName = expression[..operatorIndex].Trim();
var valuePart = expression[(operatorIndex + operatorSpan.Length)..].Trim();
if (valuePart.Length == 0)
{
return false;
}
var properties = _propCache.GetOrAdd(entity.GetType(), static t => t.GetProperties());
PropertyInfo property = null;
for (var i = 0; i < properties.Length; ++i)
{
var p = properties[i];
if (p.CanRead && p.Name.InsensitiveEquals(propertyName))
{
property = p;
break;
}
}
if (property == null)
{
return false;
}
var propertyValue = property.GetValue(entity);
var result = AdvancedSearchUtilities.CompareValues(property.PropertyType, propertyValue, valuePart, operatorSpan);
return negate ? !result : result;
}
}