### Summary
- `GenericEntityPersistence` is now a type of `GenericPersistence`. This allows developers to serialize both entities and non-entities in the same system. 🎉
- Each `SerializationThreadWorker` now allocates 1MB of heap for serialization _permanently_. If more memory is needed, that thread will double it's memory, not to exceed increments of 64MB.
- Several bugs with serialization introduced with the pure MMF implementation have been fixed.
- `BinaryFileReader` has been added back. 🎉
- Adds `world.useMultithreadedSaves` to allow disabling threaded saves.
> [!IMPORTANT]
> **Developer Note**
> The split file serialization has been deprecated and is no longer used. We have effectively gone back to the same file writing we had before the pure MMF implementation.
> [!Warning]
> **Developer Warning**
> The `PacketThrottle` callback return value is now reversed. `true` indicates the connection is _throttled_.
### Summary
- Fixes an issue where connections get stalled forever
- Fixes an issue where the throttler is not working properly
- Removes account attack limiter
- Rewrites IP limiter
- Removes IP restrictions (they weren't used, and not practical)
- Fixes issue where IP limiter was counting before firewall was blocking.
View without whitespace:
https://github.com/modernuo/ModernUO/pull/1796/files?diff=split&w=1
### Summary
- Fixes usernames not being `Intern`ed
- Reverts methods related to getting accounts from returning `Account` to `IAccount`.
- Makes `IAccount` also `ISerializable`
- Adds `IGenericReader.ReadAccount()` and `IGenericWriter.Write(IAccount)` -> The read method supports the original serialization of username, and using `IAccount.Serial`. The write method only serializes the `Serial`.
- Exposes `ReadStringRaw()` to allow some advanced scenarios.
### Summary
Adds a generic entity persistence. This can be used to create new entity types that have a `Serial`.
Here is an example:
```cs
public class BOBEntries : GenericEntitySerialization<IBOBEntry>
{
public static void Configure()
{
Configure("BOBEntries");
}
}
```
The annotation tells the system what folder to serialize the entries to. The class/interface (`IBOBEntry`) is the root type that implements `ISerializable`.
## MAJOR CHANGE
Added a champion title system to facilitate the existing champion titles. This should make it easier to extend or create other related game content. Champion titles will be saved in a folder called _ChampionTitles_.
### Motivation
The motivation to refactor was two-folder, but mostly related to performance in two ways.
First, every player had a ChampionTitleInfo object with an array of ChamptionTitleInfo. We want to eliminate the need for this information unless a player actually uses it. This should save a considerable amount of memory.
Second, to facilitate the atrophy mechanic, the champion titles would run atrophy post-world save, adding to the time that the server is frozen. Eliminating this post-world save side effect unlocks our ability to further optimize the world save process since there are no direct side effects.
### Bugs fixed
- [X] Fixed titles getting cut off on the paperdoll
- [X] Fixed champion title not displaying overhead (OPL)
### Screenshots
<img width="216" alt="image" src="https://github.com/modernuo/ModernUO/assets/3953314/8916f895-8d68-4fb0-892e-108a0c43be90">
### Changes
* Implements an AfterSerialize method that is executed synchronously.
* Removes `BeforeSerialize` support since it was dangerous in its current implementation.
* Moves PlayerMobile kill/virtual decay to AfterSerialize.
* Adds kill decay to after Deserialize.
## Changes
* Improves type hashing by introducing xxHash3 (64bit)
* Removes individual `tdb` files in favor of a single `SerializedTypes.db` file. This file is only used to identify a type that is being deserialized, which doesn't exist.
* Adds duplicate type alias detection
* Adds `AssemblyHandler.FindTypeByHash`
View changed files whitespaces: https://github.com/modernuo/ModernUO/pull/1172/files?diff=split&w=1
## SerializedTypes.db
The serialized types file is used to get back the original name of a type in case it no longer exists in code. This can easily be necessary if a class is renamed in code and no `TypeAlias` is provided.
### Format
byte[4] - version
byte[4] - count
--array--
byte[8] - xxHash
byte[1] - flag, 0 - null, 1 - not null
byte[n] - Full class name in UTF8
### Example
<img width="472" alt="SerializedTypes_Example" src="https://user-images.githubusercontent.com/3953314/195255429-31d24293-6bd1-419e-811b-07874dd0f78d.png">
## Benchmarks
Serialized 500 Type fields. The 8192bytes comes from the _ConcurrentQueue_ that would later be used for SerializedTypes.
Note that the queue is never cleared, so it's size grew considerably.
```cs
| Method | Mean | Error | StdDev | Allocated |
|--------------------- |---------:|---------:|---------:|----------:|
| BenchmarkXXHash | 18.44 us | 0.278 us | 0.260 us | 8192 B |
| BenchmarkTypeStrings | 25.09 us | 0.292 us | 0.259 us | - |
```
TODO:
* Add support in the Serialization Generator for `ReadType()` and `Write(Type)`
* Remove `SetTypeRef` from Serialization Generator
* Adds `SerializationProperty` - This will generate the private variable for serialization. Also provides an option `useField` to specify your own. Example:
```cs
[SerializationProperty(0, useField: nameof(_resource)]
```
- [X] Fixes NPE from account tags.
- [X] Fixes bad skill check due to missing cast to double.
- [X] Fixes water elemental duration.
- [X] Standardizes spell summon duration by expansion.
* Makes EnsureDirectory properly work for relative and absolute paths
* Adds a `PathUtility.GetFullPath` which returns full paths for relative paths to `Core.BaseDirectory`. If the path is absolute, it will return as-is.
* Moves EnsureDirectory to `PathUtility`. So `ScriptsHandler.EnsureDirectory` and `AssemblyHandler.EnsureDirectory` are now `PathUtility.EnsureDirectory`
* Fixes crash guard so that it copies accounts properly.
* Changes world save and auto archive to use a random folder name inside of the temp folder.
* Adds `BeforeSerialized` for entities to handle cleanup.
* Adds `Created` and `LastSerialized` fields to all entities. While this is a big bloat, this will be necessary for identifying dangling references to other invalid entities.
* Changes formula for determining a valid reference to be _not null, not deleted, and reference's created date must be at or before the entities last serialized date_.
* Adds versioning to idx file and serializes `Created` and `LastSerialized`.
* Fixes Save Stats and also disables it by default.
* Adds save flag support (see `ElvenGlasses` for an example)
* Updates AOSAttributes so they are code genned
* Fixes embedded object support by adding an `IRawSerializable`
* Fixes various inconsistencies in serializing with codegen
* Fixes pooled timer leaking
* Fixes `[dumptimers` command so it outputs properly, adds spacing, and stacktraces
* Adds `[Tidy]` for serializing Lists. This will remove deleted entities during world save before serializing the list.
* Adds helpers for managing Lists/Sets/Dictionaries
### New API
```cs
// Creates the list if it is null, then adds
Utility.Add(ref list, value);
Utility.Add(ref set, value);
Utility.Add(ref dict, key, value);
// Nulls the variable if the count is zero
Utility.Remove(ref list, value);
Utility.Remove(ref set, value);
Utility.Remove(ref dict, key);
// Marks entity as dirty in addition to doing the action
entity.Add(list, value);
// Marks entity as dirty, and will create list if it doesn't exist
entity.Add(ref list, value);
// Marks entity as dirty in addition to doing the action
entity.Remove(list, value);
// Marks entity as dirty, and will null the list count is zero
entity.Remove(ref list, value);
```
### Updates to [dumptimers
<img width="825" alt="Screen Shot 2021-08-14 at 2 55 10 AM" src="https://user-images.githubusercontent.com/3953314/129442449-ccf7fe14-29d6-4f3f-9366-c8eb7b9828a7.png">
### Changes/Fixes:
* Adds timer pooling.
* Allows pool to be configurable in ModernUO.json
* Pool replenishes itself asynchronously if depleted.
* Fixes an issue with barkeeps and town criers
* Fixes an issue with incognito buff icons not being removed
* Fixes an issue with polymorph name mod not being removed
* Fixes several places where timers go on forever even after an object is deleted, keeping a reference (memory leak)
* Eliminates the timer for MiningCart altogether.
* Deletes `AcidSlime` since it is a duplicate of `PoolOfAcid`
* Fixes HonorableExecution and standardizes the code for other Bushido moves.
## Changes to the Timer API:
```cs
public class Timer
{
// Creates a timer that will be returned to the pool once execution stops.
public static void StartTimer(Action callback);
public static void StartTimer(TimeSpan delay, Action callback);
public static void StartTimer(TimeSpan delay, TimeSpan interval, Action callback);
public static void StartTimer(TimeSpan interval, int count, Action callback);
public static void StartTimer(TimeSpan delay, TimeSpan interval, int count, Action callback);
// Creates a timer and returns a token for more control. Requires manual cancellation in order for the timer to be returned to the pool.
// If the token is dereferenced, the timer will be dereferenced too. While not returning a timer to the pool is not considered hazardous, it does defeat the purpose of pooled timers.
public static void StartTimer(Action callback, out TimerExecutionToken token);
public static void StartTimer(TimeSpan delay, Action callback, out TimerExecutionToken token);
public static void StartTimer(TimeSpan delay, TimeSpan interval, Action callback, out TimerExecutionToken token);
public static void StartTimer(TimeSpan interval, int count, Action callback, out TimerExecutionToken token);
public static void StartTimer(TimeSpan delay, TimeSpan interval, int count, Action callback, out TimerExecutionToken token);
// If you aren't sure how to use the API above, or you don't care about performance, then you can use the old RunUO Timer.DelayCall
public static DelayCallTimer DelayCall(Action callback);
public static DelayCallTimer DelayCall(TimeSpan delay, Action callback);
public static DelayCallTimer DelayCall(TimeSpan delay, TimeSpan interval, Action callback);
public static DelayCallTimer DelayCall(TimeSpan interval, int count, Action callback);
public static DelayCallTimer DelayCall(TimeSpan delay, TimeSpan interval, int count, Action callback);
}
public struct TimerExecutionToken
{
public bool Running { get; }
public int Index { get; }
public int RemainingCount { get; }
public DateTime Next { get; }
}
```
## When to use `TimerExecutionToken`?
Use tokens when you want to gain the performance benefit of using a pooled timer, but you need one of the following:
* Access to the next time the timer will tick:`token.Next`
* Access to which interval, how many intervals there are, or how many are remaining: `token.Index`, `token.Count`, and `token.RemainingCount`
* Stop a timer manually.
* Determine if the timer is running: `timer.Running`
* See notes below about requirements for using tokens!
## Notes about using the TimerExecutionToken:
When you opt-in to receive a token, you must call `Cancel()` to return the timer. This can be done inside of the callback, or outside of the callback at any time.
If this is not called and your timer is an infinite interval, then you will create a potential memory leak, or null pointer exception in your callback.
If the timer ends and is stopped, but cancel is not called, then the timer will never return to the pool and stay referenced until the token is deleted or cancel is called. (Memory leak)
## Is this thread safe?
No. The ModernUO timer system is not thread safe at all. If you require a thread safe timer system, contact me and I'll help adapt this system. Keep in mind that there is a massive performance hit to make this thread safe when there are literally no use cases for it.
If you need to synchronize execution, meaning you want to execute code from another thread on the core thread. Let's say you have a discord bot that is pushing commands to the game server. Then use `EventLoopContext.Post(SendOrPostCallback callback, object state);`.
- [X] Fixes bad `ReadEnum` by size
- [X] Fixes array, list, and set not handling null values properly.
- It will be up to the user (for now) to null out empty lists using `[AfterDeserialization]`. Convenience may be added later.
- [X] Fixes errors with `dotnet clean` and non-empty generation folder
- [X] Fixes bad field indexes on `Account.cs` causing `tags` to not be serialized/deserialized.
- This was caused by a duplicate entry. Don't have protection against this _yet_.
- [X] Fixes TimeSpan not working with codegen
- [X] Fixes bad check for generic classes with a serialize method and deserialize ctor
- [X] Moves Accounts to codegen so it is versioned
- [X] Fixes deserialization of old Accounts with no version variable.
- [X] Fixes deserialize seek not doing anything. 🙈
- [X] Adds Email to serialization
- Adds `[ManualDirtyChecking]` for anyone that adds it themselves.
- Adds detection of `[Serializable]` and `[ManualDirtyChecking]` at startup to help identify scripts that need migration.
### Additions
- Automatically opts-out `Item/Mobile/Guild/Accounts` from dirty checking with a new property `UseDirtyChecking`
- Codegen now enables `UseDirtyChecking` via getter. This requires that the property is `virtual` for derived types.
### Fixes
- Fixes `EncodedInt` being broken
- Fixes issues with new custom serializable types that are not derived from Item/Mobile/etc.
- Removes double dirty checking.
### Example of a brand new serializable type that isn't an Item/Mobile/etc.
User created code:
```cs
using System;
namespace Server.Items
{
[Serializable(0)]
public partial class NewTestEntityObject : ISerializable
{
[EncodedInt]
[SerializableField(0)]
private int _someProperty;
public NewTestEntityObject()
{
SetTypeRef(GetType());
// Add to serial tracking like World.Item
/*
Serial = World.NewEntity;
World.AddEntity(this);
*/
}
[AfterDeserialization]
private void AfterDeserialization()
{
Console.WriteLine("This ran!");
}
public int TypeRef { get; }
public Serial Serial { get; }
public void Delete()
{
}
public bool Deleted { get; set; }
public void SetTypeRef(Type type)
{
// Type tracking for persistence goes here
/*
TypeRef = World.NewEntityTypes.IndexOf(type);
if (TypeRef == -1)
{
World.NewEntityTypes.Add(type);
TypeRef = World.NewEntityTypes.Count - 1;
}
*/
}
}
}
```
Generated code:
```cs
namespace Server.Items
{
public partial class NewTestEntityObject
{
#pragma warning disable 0414
private const int _version = 0;
#pragma warning restore 0414
public int SomeProperty
{
get => _someProperty;
set
{
if (value != _someProperty)
{
_someProperty = value;
((ISerializable)this).MarkDirty();
}
}
}
long ISerializable.SavePosition { get; set; } = -1;
BufferWriter ISerializable.SaveBuffer { get; set; }
bool ISerializable.UseDirtyChecking => true;
public NewTestEntityObject(Serial serial)
{
Serial = serial;
SetTypeRef(typeof(NewTestEntityObject));
}
public void Serialize(IGenericWriter writer)
{
writer.WriteEncodedInt(_version);
writer.WriteEncodedInt(SomeProperty);
}
public void Deserialize(IGenericReader reader)
{
var version = reader.ReadEncodedInt();
SomeProperty = reader.ReadEncodedInt();
Timer.DelayCall(AfterDeserialization);
}
}
}
```
### Features
* Fully abstracts serialization by using compile-time attributes.
* Supports serializing the following:
- Primitives (integers, strings, etc)
- IP Addresses
- BigDecimal
- DateTime, Delta DateTimes
- TimeSpan
- Server.Race
- Server.Map
- Point2D, Point3D, Rect2D, Rect3D
- Existing/New `ISerializable` references
- Lists/Sets of serializable types
- Type with a `Serialize` method and constructor that takes an `IGenericReader`
* Supports forward-only migration
* Supports existing RunUO deserialization for older versions by changing to the following signature:
- `public void OldDeserialize(IGenericReader reader, int version)`
- Must remove deserializing the version since this is already done
* Supports serializing from private fields or custom made properties.
* Types do not require inheriting Item/Mobile. Code gen will fully create `ISerializable` information.
- This is not recommended yet, since it requires wiring to `Persistence` which will cause lots of unresolved symbol errors until code gen is built.
### Example
```cs
using System.Collections.Generic;
namespace Server.Items
{
[Serializable(1)]
public partial class TestItem1 : Item
{
[SerializableField(1)]
[SerializableFieldAttr("[CommandProperty(AccessLevel.Administrator)]")]
private List<Item> _someProperty;
private void Deserialize(IGenericReader reader, int version)
{
}
}
}
```
Generates this:
```cs
namespace Server.Items
{
public partial class TestItem1
{
#pragma warning disable 0414
private const int _version = 1;
#pragma warning restore 0414
[CommandProperty(AccessLevel.Administrator)]
public System.Collections.Generic.List<Server.Item> SomeProperty
{
get => _someProperty;
set
{
if (value != _someProperty)
{
((ISerializable)this).MarkDirty();
_someProperty = value;
}
}
}
public TestItem1(Serial serial) : base(serial)
{
}
public override void Serialize(IGenericWriter writer)
{
var savePosition = ((Server.ISerializable)this).SavePosition;
if (savePosition > -1)
{
writer.Seek(savePosition, System.IO.SeekOrigin.Begin);
return;
}
writer.WriteEncodedInt(_version);
writer.Write(_someProperty);
}
public override void Deserialize(IGenericReader reader)
{
var version = reader.ReadEncodedInt();
if (version < 1)
{
OldDeserialize(reader, version);
((Server.ISerializable)this).MarkDirty();
return;
}
SomeProperty = reader.ReadEntityList<Server.Item>();
}
}
}
```
And this:
```json
{
"version": 1,
"type": "TestItem1",
"properties": [
{
"name": "SomeProperty",
"type": "System.Collections.Generic.List\u003CServer.Item\u003E",
"rule": "ListMigrationRule",
"ruleArguments": [
"Server.Item",
"SerializableInterfaceMigrationRule"
]
}
]
}
```
- [X] Caches DateTime.NowUtc on the game loop (not other threads)
- [X] Replaces all locations where it makes sense
- [X] Adds Min/Max for `IComparable` (TimeSpan, DateTimes, etc)
Closes#261
- [X] Removes some string allocations (e.g. split)
- [X] Optimizes some collections
- [X] Converts insensitive to extension methods of built-ins.
- [X] Adds ordinal (case sensitive) string helpers
- [X] Fixes conditionals for in-game commands so they use Ordinal comparisons.
- [X] Replaces ToLower.Contains with InsensitiveContains
- [X] Adds ValueStringBuilder
- [X] Implements ValueStringBuilder in a few places where it makes sense
- [X] Removes the redundant Wrap function and replaces it with an optimized version
- [X] Fixes list conversions in Utility
Closes#351
Bumps release version
## Breaking Change
* IP Matching no longer supports `?` (e.g. you cannot do: "192.16?.0.1")
* IP Matching no longer supports `*` and other values in the same section (e.g. you cannot do "192.1\*.0.1")
* To do "192.1\*.0.1", you should use the range option with three separate entries: "192.100-199.0.1", "192.1.0.1", "192.10-19.0.1"
## Non-Breaking Changes
- [X] Adds IPv6 support (not for servers though, just clients)
- [X] Adds interning support for IPv4 mapped to IPv6
- [X] Updates IPv4ToAddress (don't use this unless you know it is IPv4 or IPv4 mapped to IPv6)
## Note:
UO Does not support IPv6 for the server. To support an IPv6 server IP you will need to use CUO or some kind of custom client and probably modify it accordingly.
Bumps release version
- [X] Fixes several bugs
- [X] Updates more ordinal issues
- [X] Cleans up the code a bit
- [X] Turns classes static that should have been
- [X] Changes TcpServer.Instances to a HashSet
Bumps release version
