ModernUO/Projects/Server/PropertyList/ObjectPropertyList.cs
Kamron Batman 93e46a88b2
fix: OPL revision hash collided on reordered and repeated properties (#2615)
## The bug

`ObjectPropertyList.AddHash` folded each cliloc and a Marvin hash of each argument together with XOR, which is both commutative and self-inverse. Any value mixed in an even number of times cancelled outright, so two properties sharing one argument hashed identically no matter what that argument was:

```
_hash = 31659021
AddHash(1063752); AddHash(hash("10"))
AddHash(1063737); AddHash(hash("10"))   // the argument cancels here
AddHash(1063740)                        // -> 32714560, for ANY argument
```

The 10% and 5% variants of the same item therefore produced the same revision. The client caches the tooltip by revision and only re-requests when it changes, so it kept rendering the stale percentage.

Same root cause, three more shapes:

| | old | new |
|---|---|---|
| Two properties sharing an argument | collides | distinct |
| Properties reordered | collides | distinct |
| Two properties trading arguments | collides | distinct |
| Argument-less property added twice | cancels to 0 | distinct |

## The fix

Hash the finished property block in `Terminate()` instead of accumulating per property. Those bytes — cliloc, length prefix and UTF-16 text, in emission order — are the exact content the client renders, so anything that changes the tooltip changes the hash. The length prefix also removes the concatenation ambiguity the old scheme had.

`AddHash` and both `string.GetHashCode` calls are gone.

## Why 26 bits, and why not 64

The client never recomputes the hash — it stores what we send in 0xD6 and compares it for equality against the 0xDC revision (`ObjectPropertiesListManager.IsRevisionEquals`). So the algorithm is ours to choose, but the width is not:

- Both packets carry a **4-byte** revision, so 64 bits is not available. A wider internal value would be worse than useless: the server would see a change and send a 0xDC whose truncated 32 bits are identical, and the client still wouldn't refresh.
- `Terminate` writes the bare hash into 0xD6 while `SendOPLInfo` writes `Hash` with bit 30 set. The client recovers one from the other by masking off `0x40000000`, which only holds while the hash stays below that bit. Hence 26 bits, unchanged from before.
- `Hash => 0x40000000 + _hash` keeps the revision non-zero; the client parks 0 as its "nothing cached" sentinel.

## Collision behaviour

Enumerated real small-OPL spaces and counted 26-bit collisions against the birthday expectation for a uniform hash:

| Scenario | block | n | collisions | expected |
|---|---|---|---|---|
| 1 cliloc, no arg (every cliloc 1.0M–3.2M) | 6 B | 2,200,000 | 35,591 | ~36,061 |
| 1 cliloc + numeric arg 0–199,999 | 8–12 B | 200,000 | 300 | ~298 |
| 2 clilocs, both with numeric args | 12–24 B | 202,500 | 288 | ~306 |
| 1 cliloc + 1-char arg | 8 B | 4,000,000 | 116,291 | ~119,209 |

Every case lands on the random-model line, and per-bit P(1) across all 26 bits is 0.4962–0.5024 — XXH3's short-input paths still avalanche fully, so a 6-byte block behaves like a uniform 26-bit draw. Masking the low 26 bits versus folding all 64 down was a wash (35,591 vs 35,558).

The metric that matters is narrower, since the client compares a revision only against the previous revision **for the same serial**: 2^-26 = 1.5e-8 per genuine tooltip change. Consecutive-value transitions (charges 50 -> 49) collided 0 times in 200,000.

Row 3 is the old scheme quantified: it collapsed 202,500 inputs into 100,954 distinct hashes, a 50% collision rate — systematic, not probabilistic.

One honest regression: in row 1 the old scheme had zero collisions, because for a single argument-less cliloc under 2^26 the hash was the identity function. An item whose whole tooltip is one argument-less cliloc changing to a different one goes from never colliding to 1.5e-8.

## Performance

Cheaper, not just correct — one xxHash3 pass replaces a Marvin hash per string property over those same bytes. A 12-property, 302-byte tooltip, steady state:

```
old (XOR + Marvin per property)   70.5 ns
xxHash3 (streaming, HashUtility)  27.0 ns
```

`XxHash3.HashToUInt64` one-shot is a further ~7ns faster and produces byte-identical output, but taking it would mean editing `HashUtility.ComputeHash64`, whose values are baked into save files via `AssemblyHandler.GetTypeHash`. Not worth it.

## Second commit: plant old-client property list

Found while tracing the `Hash` read sites. `PlantItem.OldClientPropertyList` called `InitializePropertyList` on every access instead of only when the list was null, and without a `Reset`. Every read appended another copy of every property to the same buffer. `SendOPLPacketTo` and `SendPropertiesTo` both go through the getter, so a pre-7.0.12 client looking at a plant grew the buffer without bound and moved the revision on reads alone.

Now builds once, matching `Item.PropertyList`. `InvalidateProperties` already `Reset`s before rebuilding.

## Testing

`Server.Tests` 869 passed, `UOContent.Tests` 779 passed.

New regression tests, each confirmed failing against the old code first:

- `RepeatedArgument_DoesNotCancelOut` — the reported case
- `PropertyOrder_ChangesHash`, `SwappedArguments_ChangeHash`, `DuplicateProperty_ChangesHash`
- `ShortNumericArguments_ConsecutiveValuesDiffer`, `SmallPropertyBlocks_StayWellDistributed` — short-input avalanche, bounded loosely enough to hold for any seed
- `Hash_StaysWithinTheRevisionMask`, `EmptyList_IsNonZeroAndDistinctFromPopulated` — wire constraints
- `PlantItemPropertyListTests.OldClientPropertyList_BuildsOnceAndIsStableAcrossReads`
2026-09-06 21:32:12 -07:00

651 lines
19 KiB
C#

/*************************************************************************
* ModernUO *
* Copyright 2019-2026 - ModernUO Development Team *
* Email: hi@modernuo.com *
* File: ObjectPropertyList.cs *
* *
* This program is free software: you can redistribute it and/or modify *
* it under the terms of the GNU General Public License as published by *
* the Free Software Foundation, either version 3 of the License, or *
* (at your option) any later version. *
* *
* You should have received a copy of the GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
*************************************************************************/
#nullable enable
using System;
using System.Buffers;
using System.Diagnostics;
using System.IO;
using System.Runtime.CompilerServices;
using Server.Buffers;
using Server.Logging;
using Server.Network;
using Server.Text;
namespace Server;
public sealed class ObjectPropertyList : IPropertyList, IDisposable
{
// Each of these are localized to "~1_NOTHING~" which allows the string argument to be used
private static readonly int[] _stringNumbers =
{
1042971,
1070722,
1114057, // ~1_val~
1114778, // ~1_val~
1114779 // ~1_val~
};
private static readonly ILogger logger = LogFactory.GetLogger(typeof(ObjectPropertyList));
// Max characters for a SINGLE OPL property argument. The legacy 2D client copies each
// property's text into a fixed ~512-char (1024-byte) buffer; exceeding it corrupts the heap
// (smashes an adjacent world object's vtable -> client crash). 504 = multiple of 8, safely
// under the empirically confirmed ~510-char ceiling. For multi-line content use AddChunked().
public const int MaxArgumentLength = 504;
// 0xD6 header: packet id, length, unknown, serial, unknown, hash. Properties start after it.
private const int HeaderLength = 15;
// Terminate writes the bare hash into 0xD6, SendOPLInfo writes Hash with bit 30 set, and the
// client recovers one from the other by masking off 0x40000000. The hash must stay below it.
private const int HashMask = 0x3FFFFFF;
private int _hash;
private int _stringNumbersIndex;
private byte[] _buffer;
private int _bufferPos;
// For string interpolation
private int _pos;
private char[]? _arrayToReturnToPool;
/// <summary>
/// True while GetProperties is populating this list. Set by the owning entity so a nested
/// InvalidateProperties can be refused instead of Reset()ing a build already in flight.
/// </summary>
internal bool IsBuilding { get; set; }
public ObjectPropertyList(IEntity? e)
{
Entity = e;
_buffer = GC.AllocateUninitializedArray<byte>(64);
var writer = new SpanWriter(_buffer);
writer.Write((byte)0xD6); // Packet ID
writer.Seek(2, SeekOrigin.Current);
writer.Write((ushort)1);
writer.Write(e?.Serial ?? Serial.Zero);
writer.Write((ushort)0);
_bufferPos = writer.Position + 4; // Hash
}
public IEntity? Entity { get; }
public int Hash => 0x40000000 + _hash;
public int Header { get; set; }
public string HeaderArgs { get; set; }
public static bool Enabled { get; set; }
public byte[] Buffer => _buffer;
public void Reset()
{
_bufferPos = HeaderLength;
_hash = 0;
_stringNumbersIndex = 0;
Header = 0;
HeaderArgs = null;
_pos = 0;
Dispose();
}
private void Flush()
{
Resize(_buffer.Length * 2);
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void Resize(int amount)
{
var newBuffer = GC.AllocateUninitializedArray<byte>(amount);
_buffer.AsSpan(0, Math.Min(amount, _buffer.Length)).CopyTo(newBuffer);
_buffer = newBuffer;
}
public void Terminate()
{
var length = _bufferPos + 4;
if (length != _buffer.Length)
{
Resize(length);
}
// xxHash3 over the finished property block. Order and repetition sensitive, unlike the
// XOR fold it replaces, which collided whenever properties were reordered or shared an
// argument.
_hash = (int)(HashUtility.ComputeHash64(_buffer.AsSpan(HeaderLength, _bufferPos - HeaderLength)) & HashMask);
var writer = new SpanWriter(_buffer);
writer.Seek(_bufferPos, SeekOrigin.Begin);
writer.Write(0);
writer.Seek(11, SeekOrigin.Begin);
writer.Write(_hash);
writer.WritePacketLength();
}
public void Add(int number)
{
if (number == 0)
{
return;
}
if (Header == 0)
{
Header = number;
HeaderArgs = "";
}
var length = _bufferPos + 6;
while (length > _buffer.Length)
{
Flush();
}
var writer = new SpanWriter(_buffer.AsSpan(_bufferPos));
writer.Write(number);
writer.Write((ushort)0);
_bufferPos += 6;
}
public void Add(int number, string? arguments) => InternalAdd(number, $"{arguments}");
public void Add(int number, int value) => InternalAdd(number, $"{value}");
public void AddLocalized(int value) => InternalAdd(GetStringNumber(), $"{value:#}");
public void AddLocalized(int number, int value) => InternalAdd(number, $"{value:#}");
public void Add(ReadOnlySpan<char> argument) => InternalAdd(GetStringNumber(), argument);
public void Add(int number, ReadOnlySpan<char> argument) => InternalAdd(number, argument);
public OplTextBlock TextBlock() => new(this);
// Emits newline-joined text across as many OPL properties as needed, breaking ONLY at '\n',
// so no single property exceeds MaxArgumentLength characters. Each chunk goes through the
// passthrough-cliloc rotation. Use for variable-length multi-line content instead of one
// Add(joined) call, which would overflow the legacy 2D-client per-property tooltip buffer.
public void AddChunked(ReadOnlySpan<char> text)
{
if (text.IsEmpty)
{
return;
}
var chunkStart = 0;
var searchFrom = 0;
while (true)
{
var nl = text[searchFrom..].IndexOf('\n');
var lineEnd = nl < 0 ? text.Length : searchFrom + nl;
// If appending this line would push the current chunk past the cap, flush the chunk
// up to the end of the previous line (excluding its '\n') first.
if (lineEnd - chunkStart > MaxArgumentLength && searchFrom > chunkStart)
{
Add(text[chunkStart..(searchFrom - 1)]);
chunkStart = searchFrom;
continue;
}
if (nl < 0)
{
Add(text[chunkStart..]); // remainder (a single over-cap line is clamped by Add)
return;
}
searchFrom = lineEnd + 1;
}
}
// Hard backstop: never let a single property exceed the legacy client's per-property buffer.
// Callers with multi-line content should use AddChunked(); this truncates anything that slips
// through and surfaces the offending entity/cliloc.
private ReadOnlySpan<char> ClampArgument(int number, ReadOnlySpan<char> chars)
{
if (chars.Length <= MaxArgumentLength)
{
return chars;
}
logger.Warning(
"OPL property on {Entity} (cliloc {Cliloc}) is {Length} chars; truncating to {Max} to avoid legacy 2D-client tooltip-buffer overflow. Use AddChunked for multi-line text.",
Entity,
number,
chars.Length,
MaxArgumentLength
);
return chars[..MaxArgumentLength];
}
private void InternalAdd(int number, ReadOnlySpan<char> chars)
{
if (number == 0)
{
return;
}
chars = ClampArgument(number, chars);
if (Header == 0)
{
Header = number;
HeaderArgs = chars.ToString();
}
var strLength = chars.Length * 2;
var length = _bufferPos + 6 + strLength;
while (length > _buffer.Length)
{
Flush();
}
var writer = new SpanWriter(_buffer.AsSpan(_bufferPos));
writer.Write(number);
writer.Write((ushort)strLength);
writer.Write(chars, TextEncoding.UnicodeLE);
_bufferPos += writer.BytesWritten;
}
private int GetStringNumber() => _stringNumbers[_stringNumbersIndex++ % _stringNumbers.Length];
// String Interpolation
public void Add(
[InterpolatedStringHandlerArgument("")]
ref IPropertyList.InterpolatedStringHandler handler
) => InternalAdd(GetStringNumber(), ref handler);
public void Add(
int number,
[InterpolatedStringHandlerArgument("")]
ref IPropertyList.InterpolatedStringHandler handler
) => InternalAdd(number, ref handler);
private void InternalAdd(
int number,
[InterpolatedStringHandlerArgument("")]
ref IPropertyList.InterpolatedStringHandler handler)
{
if (number == 0)
{
return;
}
var chars = ClampArgument(number, _arrayToReturnToPool.AsSpan(0, _pos));
if (Header == 0)
{
Header = number;
HeaderArgs = chars.ToString();
}
var strLength = chars.Length * 2;
var length = _bufferPos + 6 + strLength;
while (length > _buffer.Length)
{
Flush();
}
var writer = new SpanWriter(_buffer.AsSpan(_bufferPos));
writer.Write(number);
writer.Write((ushort)strLength);
writer.Write(chars, TextEncoding.UnicodeLE);
_bufferPos += writer.BytesWritten;
}
public void InitializeInterpolation(int literalLength, int formattedCount)
{
_arrayToReturnToPool ??= STArrayPool<char>.Shared.Rent(GetDefaultLength(literalLength, formattedCount));
_pos = 0;
}
// Copied from RawInterpolatedStringHandler
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static int GetDefaultLength(int literalLength, int formattedCount) =>
Math.Max(256, literalLength + formattedCount * 11);
// Reset()/Dispose() return the scratch buffer to the pool. If either lands while a `$"..."`
// handler is still appending, re-rent rather than spanning a null array and throwing out of
// GetProperties. Mobile/Item hold the primary guard; this covers any other caller.
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void EnsureInterpolationBuffer()
{
if (_arrayToReturnToPool == null)
{
_arrayToReturnToPool = STArrayPool<char>.Shared.Rent(256);
_pos = 0;
}
}
public void AppendLiteral(string value)
{
EnsureInterpolationBuffer();
if (value.Length == 1)
{
var chars = _arrayToReturnToPool.AsSpan();
var pos = _pos;
if ((uint)pos < (uint)chars.Length)
{
chars[pos] = value[0];
_pos = pos + 1;
}
else
{
GrowThenCopyString(value);
}
return;
}
AppendStringDirect(value);
}
private void AppendStringDirect(string value)
{
if (value.TryCopyTo(_arrayToReturnToPool.AsSpan(_pos..)))
{
_pos += value.Length;
}
else
{
GrowThenCopyString(value);
}
}
public void AppendFormatted<T>(T value)
{
EnsureInterpolationBuffer();
string? s;
if (value is IFormattable)
{
if (value is ISpanFormattable)
{
int charsWritten;
while (!((ISpanFormattable)value).TryFormat(_arrayToReturnToPool.AsSpan(_pos..), out charsWritten, default, null))
{
Grow();
}
_pos += charsWritten;
return;
}
s = ((IFormattable)value).ToString(format: null, null);
}
else
{
s = value?.ToString();
}
if (s is not null)
{
AppendStringDirect(s);
}
}
public void AppendFormatted<T>(T value, string? format)
{
EnsureInterpolationBuffer();
// '#' marks an integer argument as a cliloc ("#<value>"). Integers only -- a float/double/decimal
// '#' is the standard numeric format, not a cliloc marker.
if (format == "#" && value is int or uint or long or ulong or short or ushort or byte or sbyte)
{
AppendLiteral("#");
format = null;
}
string? s;
if (value is IFormattable)
{
if (value is ISpanFormattable)
{
int charsWritten;
while (!((ISpanFormattable)value).TryFormat(_arrayToReturnToPool.AsSpan(_pos..), out charsWritten, format, null))
{
Grow();
}
_pos += charsWritten;
return;
}
s = ((IFormattable)value).ToString(format, null);
}
else
{
s = value?.ToString();
}
if (s is not null)
{
AppendStringDirect(s);
}
}
public void AppendFormatted<T>(T value, int alignment)
{
var startingPos = _pos;
AppendFormatted(value);
if (alignment != 0)
{
AppendOrInsertAlignmentIfNeeded(startingPos, alignment);
}
}
public void AppendFormatted<T>(T value, int alignment, string? format)
{
var startingPos = _pos;
AppendFormatted(value, format);
if (alignment != 0)
{
AppendOrInsertAlignmentIfNeeded(startingPos, alignment);
}
}
public void AppendFormatted(ReadOnlySpan<char> value)
{
EnsureInterpolationBuffer();
if (value.TryCopyTo(_arrayToReturnToPool.AsSpan(_pos..)))
{
_pos += value.Length;
}
else
{
GrowThenCopySpan(value);
}
}
public void AppendFormatted(ReadOnlySpan<char> value, int alignment = 0, string? format = null)
{
EnsureInterpolationBuffer();
var leftAlign = false;
if (alignment < 0)
{
leftAlign = true;
alignment = -alignment;
}
var paddingRequired = alignment - value.Length;
if (paddingRequired <= 0)
{
AppendFormatted(value);
return;
}
EnsureCapacityForAdditionalChars(value.Length + paddingRequired);
var chars = _arrayToReturnToPool.AsSpan();
if (leftAlign)
{
value.CopyTo(chars[_pos..]);
_pos += value.Length;
chars.Slice(_pos, paddingRequired).Fill(' ');
_pos += paddingRequired;
}
else
{
chars.Slice(_pos, paddingRequired).Fill(' ');
_pos += paddingRequired;
value.CopyTo(chars[_pos..]);
_pos += value.Length;
}
}
public void AppendFormatted(string? value)
{
EnsureInterpolationBuffer();
if (value?.TryCopyTo(_arrayToReturnToPool.AsSpan(_pos..)) == true)
{
_pos += value.Length;
}
else
{
AppendFormattedSlow(value);
}
}
[MethodImpl(MethodImplOptions.NoInlining)]
private void AppendFormattedSlow(string? value)
{
if (value is not null)
{
EnsureCapacityForAdditionalChars(value.Length);
value.CopyTo(_arrayToReturnToPool.AsSpan(_pos..));
_pos += value.Length;
}
}
public void AppendFormatted(string? value, int alignment = 0, string? format = null) =>
AppendFormatted<string?>(value, alignment, format);
public void AppendFormatted(object? value, int alignment = 0, string? format = null) =>
AppendFormatted<object?>(value, alignment, format);
private void AppendOrInsertAlignmentIfNeeded(int startingPos, int alignment)
{
Debug.Assert(startingPos >= 0 && startingPos <= _pos);
Debug.Assert(alignment != 0);
var charsWritten = _pos - startingPos;
var leftAlign = false;
if (alignment < 0)
{
leftAlign = true;
alignment = -alignment;
}
var paddingNeeded = alignment - charsWritten;
if (paddingNeeded > 0)
{
EnsureCapacityForAdditionalChars(paddingNeeded);
var chars = _arrayToReturnToPool.AsSpan();
if (leftAlign)
{
chars.Slice(_pos, paddingNeeded).Fill(' ');
}
else
{
chars.Slice(startingPos, charsWritten).CopyTo(chars[(startingPos + paddingNeeded)..]);
chars.Slice(startingPos, paddingNeeded).Fill(' ');
}
_pos += paddingNeeded;
}
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void EnsureCapacityForAdditionalChars(int additionalChars)
{
if (_arrayToReturnToPool.Length - _pos < additionalChars)
{
Grow(additionalChars);
}
}
[MethodImpl(MethodImplOptions.NoInlining)]
private void GrowThenCopyString(string value)
{
Grow(value.Length);
value.CopyTo(_arrayToReturnToPool.AsSpan(_pos..));
_pos += value.Length;
}
[MethodImpl(MethodImplOptions.NoInlining)]
private void GrowThenCopySpan(ReadOnlySpan<char> value)
{
Grow(value.Length);
value.CopyTo(_arrayToReturnToPool.AsSpan(_pos..));
_pos += value.Length;
}
[MethodImpl(MethodImplOptions.NoInlining)]
private void Grow(int additionalChars)
{
Debug.Assert(additionalChars > _arrayToReturnToPool.Length - _pos);
GrowCore((uint)_pos + (uint)additionalChars);
}
[MethodImpl(MethodImplOptions.NoInlining)]
private void Grow()
{
GrowCore((uint)_arrayToReturnToPool.Length + 1);
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void GrowCore(uint requiredMinCapacity)
{
var newCapacity = Math.Max(requiredMinCapacity, Math.Min((uint)_arrayToReturnToPool.Length * 2, 0x3FFFFFDF));
var arraySize = (int)Math.Clamp(newCapacity, 256, int.MaxValue);
var newArray = STArrayPool<char>.Shared.Rent(arraySize);
_arrayToReturnToPool.AsSpan(.._pos).CopyTo(newArray);
var toReturn = _arrayToReturnToPool;
_arrayToReturnToPool = newArray;
STArrayPool<char>.Shared.Return(toReturn);
}
public void Dispose()
{
if (_arrayToReturnToPool != null)
{
STArrayPool<char>.Shared.Return(_arrayToReturnToPool);
_arrayToReturnToPool = null;
}
}
~ObjectPropertyList()
{
if (_arrayToReturnToPool != null)
{
STArrayPool<char>.Shared.Return(_arrayToReturnToPool);
_arrayToReturnToPool = null;
}
}
}