## Summary
- Adds proper Latin1 encoding support, replacing CP1252 usage throughout the codebase
- Adds specialized, optimized string decoding methods with safe string filtering for each encoding type
- Filters invalid Unicode characters (C0/C1 control codes, non-characters) by removal rather than replacement since
the UO client renders nothing for these characters
- Fixes UTF-16 null terminator position handling to correctly advance by 2 bytes
## Changes
TextEncoding.cs
- Added SearchValues-based invalid byte/char detection for efficient filtering
- Added encoding-specific GetString methods: GetStringAscii, GetStringLatin1, GetStringUtf8, GetStringBigUni,
GetStringLittleUni
- Each method supports a safeString parameter for filtering invalid characters
- Little-endian UTF-16 uses direct memory cast for zero-copy decoding on LE systems
- Invalid characters are removed (not replaced with U+FFFD) since the client renders nothing for them
SpanReader.cs
- Added ReadLatin1() and ReadLatin1Safe() methods
- Rewrote encoding-specific read methods to use optimized TextEncoding.GetString* methods
- Fixed UTF-16 null terminator handling: position now correctly advances by byteLength (2) instead of 1
SpanWriter.cs
- Added WriteLatin1 and WriteLatin1Null methods
## Packet Updates
- Updated all packet code to use Latin1 encoding instead of CP1252
- Affected: account packets, equipment packets, menu packets, message packets, mobile packets, player packets, secure
trade packets, vendor packets, gump packets, book packets, mahjong packets
## Filtering Behavior
Invalid characters filtered in safe mode:
```
┌───────────────┬────────────────────────┐
│ Range │ Description │
├───────────────┼────────────────────────┤
│ 0x00-0x1F │ C0 control codes │
├───────────────┼────────────────────────┤
│ 0x7F │ DEL │
├───────────────┼────────────────────────┤
│ 0x80-0x9F │ C1 control codes │
├───────────────┼────────────────────────┤
│ 0xFFFE-0xFFFF │ Unicode non-characters │
└───────────────┴────────────────────────┘
```
Note: Surrogate pairs (0xD800-0xDFFF) are not filtered because proper validation requires context checking for paired
vs unpaired surrogates. The UO client renders nothing for these anyway.
## Test Plan
- All 631 Server.Tests pass
- Verified client rendering behavior using TestUnicodeGump command (pages 1-5)
- Confirmed U+FFFD, unpaired surrogates, and non-characters all render as blank in client
- Verified Latin1 characters (0xA0-0xFF) display correctly
- Verified C1 control codes (0x80-0x9F) are filtered and don't display
## Breaking Changes
Incoming packet registration signature has changed to:
```cs
delegate* void OnReceiveCallback(NetState state, SpanReader reader, int packetLength);
IncomingPackets.Register(int packetID, int length, bool ingame, OnReceiveCallback onReceive);
```
For example, an incoming packet handler signature would now look like this:
```cs
public static void SomeIncomingPacket(NetState state, SpanReader reader, int packetLength)
{
// Parse the data
}
```
## Summary
Updates the network Pipe class to use a mirrored memory technique. This technique involves mapping the same physical memory to two contiguous virtual memory spaces so the byte buffer appears duplicated. This allows writing to a double-sized array to wrap around without the need for the `CircularBuffer` classes.
In practice this allows us to use `Span<byte>` as if the buffer was a regular array.
### Bug Fixes
- [X] Fixes bad fixed length string parsing
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
* Introduces `RawInterpolatedStringHandler` which is exactly the same as `DefaultInterpolatedStringHandler` except it _unsafely exposes_ it's `ReadOnlySpan<char>` buffer. This is useful for writing the string's data without actually building the string.
* Uses this new string interpolation handler in `SpanWriter` to eliminate intermediate strings built. This is immensely useful in eliminating string allocations in writing Gump packets.
- [X] Adds `AdhocPersistence` which replaces RunUO `Persistence`
- [X] Fixes a few minor bugs with EntityPersistence deserialization
- [X] Reverts/updates some serialization changes
- [X] Fixes map issues at New Haven by turning off static diffs
- [X] Some code cleanup and reformatting of tile matrix, tile matrix patch, and tile data
- [X] Adds NetState.Flush for generating spawners so it doesn't feel like the server is frozen
- [X] Reverts NativeReader changes from a while back.
- [X] Adds more string reading for BufferReader.
Notes:
BufferReader is still `little endian` compared to `SpanReader` which is `big endian` (for packets). To that end, the RunUO deserialization `ReadString()` was made obsolete since it is ambiguous, and contains extra fields other than simply reading a string. Furthermore, we shouldn't be using UTF8 (for now) since it is slow.
SpanWriter now has an argument that allows it to be resizable.
```cs
public SpanWriter(Span<byte> initialBuffer, bool resize = false)
public SpanWriter(int initialCapacity, bool resize = false)
```
If a SpanWriter is set to be resizable, or given an initial capacity instead of a buffer, it must be disposed:
```cs
public static void SomeMethod()
{
using var writer = new SpanWriter(stackalloc byte[512], true);
// stuff
// Automatic Dispose of writer
}
```
This is because the SpanWriter uses `ArrayPool<byte>.Shared` _rented buffers_ to resize. If the writer is not disposed, those buffers will never be reused resulting in a _memory leak_.
It is possible that the SpanWriter will outright ditch the initial buffer if resize is set to true and growing is needed. To check/account for that we can do the following:
```cs
public static void SomeMethod()
{
Span<byte> span = stackalloc byte[512];
using var writer = new SpanWriter(span, true);
// write some stuff that causes the span to grow
span = writer.RawBuffer;
// Do stuff with the span
}
```
Make sure you don't accidentally `Dispose()` the writer or use the `RawBuffer` outside of the `using` block. If you do, bad things will happen! (NullPointerException, or a fresh SpanWriter with no buffer, depending on the situation).
SpanWriter can also now be used with a fixed statement since it has a `PinnableReference()` function.
- [X] Fixes several bugs
- [X] Updates more ordinal issues
- [X] Cleans up the code a bit
- [X] Turns classes static that should have been
- [X] Changes TcpServer.Instances to a HashSet
Bumps release version
