The idx v4 type table supersedes the global type queue: workers no longer
enqueue every Write(Type) into a shared ConcurrentQueue during the freeze,
WriteFiles no longer drains and dedupes millions of entries, and the db file
is no longer produced. Loading old saves still reads SerializedTypes.db via
the legacy v2/v3 path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier-1 disasm of a generated-style Serialize showed PGO's guarded
devirtualization already inlines every IGenericWriter.Write body (interface
vs concrete-typed callsites measured identical), so no API or generator
changes are needed. What remained as a real call per primitive write was the
Index property setter - too large to inline due to its range-check throw path,
plus per-write high-water tracking, an AsSpan bounds check, and a
BinaryPrimitives span check.
The write path now reserves capacity once (the existing grow-on-Flush check),
then does an unaligned store through a ref with a raw index increment - the
capacity check proves the store in-bounds, and the index only moves forward
between Seeks. The _bytesWritten high-water mark (used only by
SeekOrigin.End) folds at Seek/Resize instead of per write, which is
equivalent because writes are monotonic between seeks. The Index property
keeps its validating semantics for Seek and subclasses.
BufferWriter also gains class-level implementations of the hottest
IGenericWriter default interface methods (WriteEncodedInt, DateTime,
TimeSpan, Point2D/3D, Rectangle2D/3D, Map, Race): a DIM dispatches again on
`this` for every nested Write even at a devirtualized callsite, and the class
overloads keep the whole write inlined.
Strings (arbitrary UTF-16) previously walked every string twice
(GetByteCount then GetBytes) because the variable-width length prefix
precedes the bytes. Strings of 85 chars or fewer (any content, incl.
surrogate pairs - 85 * 3 = 255 bytes max) now encode once into a 256-byte
stack scratch, then write the prefix and copy. Byte output is identical.
Measured: 34.4 -> 15.7 ns/entity on a generated-style write mix (~20 writes,
58 bytes), 22.5 -> 11.8 ns per short-string write, and the end-to-end freeze
benchmark (10M entities / 1.7GB / 24 cores, dense profile) drops from ~99ms
to ~74-82ms. Combined with the earlier pipeline commits: ~740ms -> ~78ms.
New tests pin byte-level output and position semantics: primitive
little-endian layouts, Seek(End) high-water behavior, growth preservation,
span writes across growth, encoded-int formats, and string equivalence
across the scratch/two-pass boundary with mixed-width UTF-16 content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
### Summary
Updates the serialization strategy to use `MemoryMappedFile` instead of thick buffers. This has the benefit of being on-par with the current implementation (based on hardware/OS), however won't incur the double-memory issue.
> [!Important]
> **Developer Note**
> The `BinaryFileWriter` and `BinaryFileReader` has been removed in favor of `MemoryMapFileWriter` and `UnmanagedDataReader`
### Summary
- Fixes infinite loop with binary file writer
- Removes extra buffer copying with binary file writer
- Removes storing type counts during world save file writing
- Fixes display cache self-deletion warning during world load
### Summary
* Fixes spawner timer deserialization
* Adds a check for a null timer and allows the timer to get recreated
* Adds PotionKeg reverse lookup
* Heavily optimizes decimal serialize/deserialize
### Summary
The BitArray class will be optimized over the next several years for various platforms/hardware and maintaining a duplicate for serialization is not practical. Removing the custom implementation. Recommend against using BitArray for serialization unless it is absolutely necessary.
### Summary
Updates BufferWriter with more standard ways of writing primitives. Eliminates looping to write a string per @jaedan's suggestion.
Note: This change assumes we don't have crazy large strings.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
**Only one functional change**
* Fixes a bug in LogFactory where `Warning` is being logged as `Information`
Non-functional changes:
* Updates/Fixes copyright headers
* Removes namespace scopes for core files.
View with [whitespace off](https://github.com/modernuo/ModernUO/pull/1187/files?w=1).
Fixes bit array serialization. This may cause objects that were serialized by bit array to fail to deserialize. I am sorry, please accept my condolences. It is probably easiest to just delete those objects. If it becomes a major problem, contact me and I'll help with a hacky per-case solution.
* Adds a custom BitArray class with the following added features:
* ctor for creating BitArray against read only span
* ctor for creating BitArray against BinaryReader
* CopyTo to copy a BitArray to a Span
* Adds BitArray to UO Primitive serialization so it can be codegenned.
### Features
* Fully abstracts serialization by using compile-time attributes.
* Supports serializing the following:
- Primitives (integers, strings, etc)
- IP Addresses
- BigDecimal
- DateTime, Delta DateTimes
- TimeSpan
- Server.Race
- Server.Map
- Point2D, Point3D, Rect2D, Rect3D
- Existing/New `ISerializable` references
- Lists/Sets of serializable types
- Type with a `Serialize` method and constructor that takes an `IGenericReader`
* Supports forward-only migration
* Supports existing RunUO deserialization for older versions by changing to the following signature:
- `public void OldDeserialize(IGenericReader reader, int version)`
- Must remove deserializing the version since this is already done
* Supports serializing from private fields or custom made properties.
* Types do not require inheriting Item/Mobile. Code gen will fully create `ISerializable` information.
- This is not recommended yet, since it requires wiring to `Persistence` which will cause lots of unresolved symbol errors until code gen is built.
### Example
```cs
using System.Collections.Generic;
namespace Server.Items
{
[Serializable(1)]
public partial class TestItem1 : Item
{
[SerializableField(1)]
[SerializableFieldAttr("[CommandProperty(AccessLevel.Administrator)]")]
private List<Item> _someProperty;
private void Deserialize(IGenericReader reader, int version)
{
}
}
}
```
Generates this:
```cs
namespace Server.Items
{
public partial class TestItem1
{
#pragma warning disable 0414
private const int _version = 1;
#pragma warning restore 0414
[CommandProperty(AccessLevel.Administrator)]
public System.Collections.Generic.List<Server.Item> SomeProperty
{
get => _someProperty;
set
{
if (value != _someProperty)
{
((ISerializable)this).MarkDirty();
_someProperty = value;
}
}
}
public TestItem1(Serial serial) : base(serial)
{
}
public override void Serialize(IGenericWriter writer)
{
var savePosition = ((Server.ISerializable)this).SavePosition;
if (savePosition > -1)
{
writer.Seek(savePosition, System.IO.SeekOrigin.Begin);
return;
}
writer.WriteEncodedInt(_version);
writer.Write(_someProperty);
}
public override void Deserialize(IGenericReader reader)
{
var version = reader.ReadEncodedInt();
if (version < 1)
{
OldDeserialize(reader, version);
((Server.ISerializable)this).MarkDirty();
return;
}
SomeProperty = reader.ReadEntityList<Server.Item>();
}
}
}
```
And this:
```json
{
"version": 1,
"type": "TestItem1",
"properties": [
{
"name": "SomeProperty",
"type": "System.Collections.Generic.List\u003CServer.Item\u003E",
"rule": "ListMigrationRule",
"ruleArguments": [
"Server.Item",
"SerializableInterfaceMigrationRule"
]
}
]
}
```
- [X] Adds `AdhocPersistence` which replaces RunUO `Persistence`
- [X] Fixes a few minor bugs with EntityPersistence deserialization
- [X] Reverts/updates some serialization changes
- [X] Caches DateTime.NowUtc on the game loop (not other threads)
- [X] Replaces all locations where it makes sense
- [X] Adds Min/Max for `IComparable` (TimeSpan, DateTimes, etc)
Closes#261
- [X] Fixes an issue in BufferWriter where if SeekOrigin.End is used, it will yield the wrong index.
- [X] Changes BinaryFileWriter to dispose pattern
- [X] Removes World.SaveBuffers. They were dangerous and should be done in-line while the world is loading, even if it is slower
- [X] Fixes BufferReader Seek too
- [X] Changes BufferWriter to use uninitialized arrays to speed up resizing
- [X] Changes World loading so it doesn't buffer the whole file, even if it makes things slower. It lowers allocations/fragmentation over all which is good.
- [X] Changes World Loading to use uninitialized arrays and directly saves them to the SaveBuffer which eliminates having to open/load the files _again_ which is dangerous. Also looping through items while the world is running is dangerous since async references can and will cause exceptions.
Bumps release version
- [X] Fixes an issue where a buffer smaller than 8 bytes would not double with enough space in some cases.
- [X] Fixes an issue with dupe copying the savebuffer reference (ugh).
- [X] Streamlines the IGenericWriter API to use better generics.
- [X] Streamlines the IGenericReader API to use better generics.
- [X] Forces `tidying` of a List/HashSet to be done externally since Writers/Readers should not have side effects.
- [X] Fixes an issue where Tidying a list didn't TrimExcess, causing memory leaks.
- [X] Reverted the meaning of `World.Running` to specifically refer to any world state post world loading.
- NOTE: Do not use this if you want to block on world saves. Instead use checks against `WorldState.Saving` states.
- [X] Fixes an issue with serializing negative DateTime deltas.
- [X] Fixes a potential issue with serializing non-UTC DateTime.
Bumps release version
- [X] Removes some string allocations (e.g. split)
- [X] Optimizes some collections
- [X] Converts insensitive to extension methods of built-ins.
- [X] Adds ordinal (case sensitive) string helpers
- [X] Fixes conditionals for in-game commands so they use Ordinal comparisons.
- [X] Replaces ToLower.Contains with InsensitiveContains
- [X] Adds ValueStringBuilder
- [X] Implements ValueStringBuilder in a few places where it makes sense
- [X] Removes the redundant Wrap function and replaces it with an optimized version
- [X] Fixes list conversions in Utility
Closes#351
Bumps release version
- [X] Fixes IPAddresses not having enough space
- [X] Streamlines some code
- [X] Fixes base escortables to not load destination tables in a dangerous way.
- [X] Fixes some bad assumptions about resizing buffers
- [X] Fixes little endian issue with circular buffer writer
This doesn't seem to address #349. That issue requires more investigation because I am not seeing where the issue would be.
Closes#347Closes#322
- [X] Speeds up world loading
- [X] Removes PeekChar cause it wasn't used
- [X] Adds a BufferReader
- [X] Removes BinaryFileReader
- [X] Removes reading/writing `char` since it is not consistent and will cause a deserialize issue if used.
Bumps version release
- [X] Fixes several bugs
- [X] Updates more ordinal issues
- [X] Cleans up the code a bit
- [X] Turns classes static that should have been
- [X] Changes TcpServer.Instances to a HashSet
Bumps release version
- [X] Fixes bugs in the buffer writer
- [X] Removes background save commands
- [X] Avoids calling Serialize() on deserialization
- In a future optimization I will copy the entire object to a buffer, deserialize using a SpanReader, then use that buffer for the SaveBuffer
- [X] Fixes issue with decay queue.
Bumps release version
- [X] Removes all save strategies
- [X] Removes duplicate file writer that won't be used
- [X] Removes persistence (it will become a duplicate system)
- [X] Add a save position variable to skip serializing a clean item/mobile
- [X] Update Guilds/Accounts to be IEntity types
- Because guilds are abstract, this may make serialization tricky.
- [X] Update the generalized IEntity writing
- [X] Update the load/save to write to buffers then to files in background
Bumps release version
