warrentyexpired/ModernUO
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: Adds ReadType/Write(Type) and improves type referencing (#1172)

Browse source 
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`

View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1

## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.

### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8

### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">

## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
|               Method |     Mean |    Error |   StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
|      BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us |    8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us |         - |
```

TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
This commit is contained in:
Kamron Batman 2022-10-11 22:17:22 -07:00 • committed by GitHub
parent f268d5d4e2
commit e1e30998ba
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
28 changed files with 616 additions and 291 deletions
Download patch file Download diff file Expand all files Collapse all files

45
Projects/Server/Serialization/AdhocPersistence.cs
View file

@ -14,6 +14,8 @@
*************************************************************************/
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Threading.Tasks;
@ -27,30 +29,53 @@ public static class AdhocPersistence
* Note: The buffer may not be the same after returning from the function if more data is written
* than the initial buffer can handle.
*/
public static BufferWriter Serialize(Action<IGenericWriter> serializer)
public static BufferWriter Serialize(Action<IGenericWriter> serializer, ConcurrentQueue<Type> types)
{
var saveBuffer = new BufferWriter(true);
var saveBuffer = new BufferWriter(true, types);
serializer(saveBuffer);
return saveBuffer;
}
/**
* Writes a buffer to disk. This function should be called asynchronously.
* Writes the filePath for the binary data, and an accompanying SerializedTypes.db file of all possible types.
*/
public static void WriteSnapshot(string filePath, Span<byte> buffer)
public static void WriteSnapshot(FileInfo file, Span<byte> buffer)
{
var fullPath = PathUtility.GetFullPath(filePath, Core.BaseDirectory);
var file = new FileInfo(fullPath);
PathUtility.EnsureDirectory(file.DirectoryName);
var dirPath = file.DirectoryName;
PathUtility.EnsureDirectory(dirPath);
using var fs = new FileStream(fullPath, FileMode.Create, FileAccess.Write);
using var fs = new FileStream(file.FullName, FileMode.Create, FileAccess.Write);
fs.Write(buffer);
}
public static void SerializeAndSnapshot(string filePath, Action<IGenericWriter> serializer)
/**
* Serializes to a memory buffer synchronously, then flushes to the path asynchronously.
* See WriteSnapshot for more info about how the snapshot.
*/
public static void SerializeAndSnapshot(string filePath, Action<IGenericWriter> serializer, ConcurrentQueue<Type> types = null)
{
var saveBuffer = Serialize(serializer);
Task.Run(() => { WriteSnapshot(filePath, saveBuffer.Buffer.AsSpan(0, (int)saveBuffer.Position)); });
types ??= new ConcurrentQueue<Type>();
var saveBuffer = Serialize(serializer, types);
Task.Run(
() =>
{
var fullPath = PathUtility.GetFullPath(filePath, Core.BaseDirectory);
var file = new FileInfo(fullPath);
WriteSnapshot(file, saveBuffer.Buffer.AsSpan(0, (int)saveBuffer.Position));
// TODO: Create a PooledHashSet if performance becomes an issue.
var typesSet = new HashSet<Type>();
// Dedupe the queue.
foreach (var type in types)
{
typesSet.Add(type);
}
Persistence.WriteSerializedTypesSnapshot(file.DirectoryName, typesSet);
});
}
public static void Deserialize(string filePath, Action<IGenericReader> deserializer)