0404251638
## Summary - Adds proper Latin1 encoding support, replacing CP1252 usage throughout the codebase - Adds specialized, optimized string decoding methods with safe string filtering for each encoding type - Filters invalid Unicode characters (C0/C1 control codes, non-characters) by removal rather than replacement since the UO client renders nothing for these characters - Fixes UTF-16 null terminator position handling to correctly advance by 2 bytes ## Changes TextEncoding.cs - Added SearchValues-based invalid byte/char detection for efficient filtering - Added encoding-specific GetString methods: GetStringAscii, GetStringLatin1, GetStringUtf8, GetStringBigUni, GetStringLittleUni - Each method supports a safeString parameter for filtering invalid characters - Little-endian UTF-16 uses direct memory cast for zero-copy decoding on LE systems - Invalid characters are removed (not replaced with U+FFFD) since the client renders nothing for them SpanReader.cs - Added ReadLatin1() and ReadLatin1Safe() methods - Rewrote encoding-specific read methods to use optimized TextEncoding.GetString* methods - Fixed UTF-16 null terminator handling: position now correctly advances by byteLength (2) instead of 1 SpanWriter.cs - Added WriteLatin1 and WriteLatin1Null methods ## Packet Updates - Updated all packet code to use Latin1 encoding instead of CP1252 - Affected: account packets, equipment packets, menu packets, message packets, mobile packets, player packets, secure trade packets, vendor packets, gump packets, book packets, mahjong packets ## Filtering Behavior Invalid characters filtered in safe mode: ``` ┌───────────────┬────────────────────────┐ │ Range │ Description │ ├───────────────┼────────────────────────┤ │ 0x00-0x1F │ C0 control codes │ ├───────────────┼────────────────────────┤ │ 0x7F │ DEL │ ├───────────────┼────────────────────────┤ │ 0x80-0x9F │ C1 control codes │ ├───────────────┼────────────────────────┤ │ 0xFFFE-0xFFFF │ Unicode non-characters │ └───────────────┴────────────────────────┘ ``` Note: Surrogate pairs (0xD800-0xDFFF) are not filtered because proper validation requires context checking for paired vs unpaired surrogates. The UO client renders nothing for these anyway. ## Test Plan - All 631 Server.Tests pass - Verified client rendering behavior using TestUnicodeGump command (pages 1-5) - Confirmed U+FFFD, unpaired surrogates, and non-characters all render as blank in client - Verified Latin1 characters (0xA0-0xFF) display correctly - Verified C1 control codes (0x80-0x9F) are filtered and don't display
54 lines
3.4 KiB
XML
54 lines
3.4 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current">
|
|
<PropertyGroup>
|
|
<Product>ModernUO Core</Product>
|
|
<OutDir>..\..\Distribution</OutDir>
|
|
<PublishDir>..\..\Distribution</PublishDir>
|
|
<Version>0.0.0</Version>
|
|
<Configurations>Debug;Release;Analyze</Configurations>
|
|
<RootNamespace>Server</RootNamespace>
|
|
<PackageId>Server</PackageId>
|
|
</PropertyGroup>
|
|
<Target Name="CleanPub" AfterTargets="Clean">
|
|
<Message Text="Server: Removing distribution files..." />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName)" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).deps.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).dll.config" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).exe" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).pdb" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).runtimeconfig.dev.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\$(AssemblyName).runtimeconfig.json" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\ModernUO.Serialization.Annotations.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\CommunityToolkit.HighPerformance.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\PollGroup.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\ref\$(AssemblyName).dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.Sinks.Async.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\Serilog.Sinks.Console.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\wepoll.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\LibDeflate.Bindings.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\libdeflate.dll" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\libdeflate.dylib" ContinueOnError="true" />
|
|
<Delete Files="..\..\Distribution\System.IO.Hashing.dll" ContinueOnError="true" />
|
|
</Target>
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\Logger\Logger.csproj" />
|
|
<PackageReference Include="CommunityToolkit.HighPerformance" Version="8.4.0" />
|
|
<PackageReference Include="LibDeflate.Bindings" Version="1.0.2.120" />
|
|
<PackageReference Include="PollGroup" Version="1.7.1" />
|
|
<PackageReference Include="System.IO.Hashing" Version="10.0.2" />
|
|
|
|
<PackageReference Include="ModernUO.Serialization.Annotations" Version="2.14.0" />
|
|
<PackageReference Include="ModernUO.Serialization.Generator" Version="2.14.1" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<AdditionalFiles Include="Migrations/*.v*.json" />
|
|
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
|
<_Parameter1>Server.Tests</_Parameter1>
|
|
</AssemblyAttribute>
|
|
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
|
<_Parameter1>UOContent.Tests</_Parameter1>
|
|
</AssemblyAttribute>
|
|
</ItemGroup>
|
|
</Project>
|