Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork747
Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#]
License
MessagePack-CSharp/MessagePack-CSharp
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
The extremely fastMessagePack serializer for C#.It is 10x faster thanMsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like games, distributed computing, microservices, or data caches.
MessagePack has a compact binary size and a full set of general purpose expressive data types. Please have a look at thecomparison with JSON, protobuf, ZeroFormatter section and learnwhy MessagePack C# is the fastest.
- Installation
- Quick Start
- Analyzer
- Built-in supported types
- Object Serialization
- DataContract compatibility
- Serializing readonly/immutable object members (SerializationConstructor)
- Serialization Callback
- Union
- Dynamic (Untyped) Deserialization
- Object Type Serialization
- Typeless
- Security
- Performance
- LZ4 Compression
- Comparison with protobuf, JSON, ZeroFormatter
- Hints to achieve maximum performance when using MessagePack for C#
- Extensions
- Experimental Features
- High-Level API (
MessagePackSerializer
) - Low-Level API (
IMessagePackFormatter<T>
) - Primitive API (
MessagePackWriter
,MessagePackReader
) - Main Extension Point (
IFormatterResolver
) - MessagePackFormatterAttribute
- IgnoreFormatter
- Reserved Extension Types
- Unity support
- AOT Code Generation (support for Unity/Xamarin)
- RPC
- How to build
- Author Info
- Code of Conduct & .NET Foundation notice
This library is distributed via NuGet. SpecialUnity support is available, too.
We target .NET Standard 2.0 with special optimizations for .NET 8+ and .NET Framework.The library code is pure C# (with Just-In-Time IL code generation on some platforms or AOT safe source generators).
To install with NuGet, just install theMessagePack
package:
Install-Package MessagePack
There are also a range of official and third party Extension Packages available (learn more in ourextensions section):
Install-Package MessagePack.ReactivePropertyInstall-Package MessagePack.UnityShimsInstall-Package MessagePack.AspNetCoreMvcFormatter
For Unity projects, please read theUnity Support section to install.
Migrating from a prior major version of MessagePack to the latest?Check outthese instructions.
Define the struct or class to be serialized and annotate it with a[MessagePackObject]
attribute.Annotate members whose values should be serialized (fields as well as properties) with[Key]
attributes.
[MessagePackObject]publicclassMyClass{// Key attributes take a serialization index (or string name)// The values must be unique and versioning has to be considered as well.// Keys are described in later sections in more detail.[Key(0)]publicintAge{get;set;}[Key(1)]publicstringFirstName{get;set;}[Key(2)]publicstringLastName{get;set;}// All fields or properties that should not be serialized must be annotated with [IgnoreMember].[IgnoreMember]publicstringFullName{get{returnFirstName+LastName;}}}
CallMessagePackSerializer.Serialize<T>/Deserialize<T>
to serialize/deserialize your object instance.You can use theConvertToJson
method to get a human readable representation of any MessagePack binary blob.
classProgram{staticvoidMain(string[]args){varmc=newMyClass{Age=99,FirstName="hoge",LastName="huga",};// Call Serialize/Deserialize, that's all.byte[]bytes=MessagePackSerializer.Serialize(mc);MyClassmc2=MessagePackSerializer.Deserialize<MyClass>(bytes);// You can dump MessagePack binary blobs to human readable json.// Using indexed keys (as opposed to string keys) will serialize to MessagePack arrays,// hence property names are not available.// [99,"hoge","huga"]varjson=MessagePackSerializer.ConvertToJson(bytes);Console.WriteLine(json);}}
By default, aMessagePackObject
annotation is required. This can be made optional; see theObject Serialization section and theFormatter Resolver section for details.
The MessagePackAnalyzer package(includes in default) aids with:
- Produces compiler warnings upon incorrect attribute use, member accessibility, and more.
- Automating attributing of your serializable classes and members.
- Optionally improving startup time throughAOT formatter generation.
The first two of these features is demonstrated below:
Two assembly-level attributes exist to help with mixing in your own custom formatters with the automatically generated ones:
MessagePackKnownFormatterAttribute
- Identifies classes that implementIMessagePackFormatter<T>
.TheT
type argument willnot produce an analyzer warning whenT
is used elsewhere in a serializable object.When using a source generated resolver, the resolver will refer to this formatter for the appropriate type(s).MessagePackAssumedFormattableAttribute
- Identifies types that are assumed to have anIMessagePackFormatter<T>
somewhere, and that will be combined within anIFormatterResolver
at runtime to ensure the specified type can be serialized.This attribute will suppress the analyzer warning from using that type although the type does not have a[MessagePackObject]
attribute on it.
These types can serialize by default:
- Primitives (
int
,string
, etc...),Enum
s,Nullable<>
,Lazy<>
TimeSpan
,DateTime
,DateTimeOffset
Guid
,Uri
,Version
,StringBuilder
BigInteger
,Complex
,Half
Array[]
,Array[,]
,Array[,,]
,Array[,,,]
,ArraySegment<>
,BitArray
KeyValuePair<,>
,Tuple<,...>
,ValueTuple<,...>
ArrayList
,Hashtable
List<>
,LinkedList<>
,Queue<>
,Stack<>
,HashSet<>
,ReadOnlyCollection<>
,SortedList<,>
IList<>
,ICollection<>
,IEnumerable<>
,IReadOnlyCollection<>
,IReadOnlyList<>
Dictionary<,>
,IDictionary<,>
,SortedDictionary<,>
,ILookup<,>
,IGrouping<,>
,ReadOnlyDictionary<,>
,IReadOnlyDictionary<,>
ObservableCollection<>
,ReadOnlyObservableCollection<>
ISet<>
,ConcurrentBag<>
,ConcurrentQueue<>
,ConcurrentStack<>
,ConcurrentDictionary<,>
- Immutable collections (
ImmutableList<>
, etc) - Custom implementations of
ICollection<>
orIDictionary<,>
with a parameterless constructor - Custom implementations of
IList
orIDictionary
with a parameterless constructor
You can add support for custom types, and there are some official/third-party extension packages for:
- ReactiveProperty
- for Unity (
Vector3
,Quaternion
, etc...) - F# (Record, FsList, Discriminated Unions, etc...)
Please see theextensions section.
MessagePack.Nil
is the built-in type representing null/void in MessagePack for C#.
MessagePack for C# can serialize your own publicclass
orstruct
types. By default, serializable types must be annotated with the[MessagePackObject]
attribute and members with the[Key]
attribute. Keys can be either indexes (int
) or arbitrary strings. If all keys are indexes, arrays are used for serialization, which offers advantages in performance and binary size. Otherwise, MessagePack maps (dictionaries) will be used.
If you use[MessagePackObject(keyAsPropertyName: true)]
, then members do not require explicitKey
attributes, but string keys will be used.
[MessagePackObject]publicclassSample1{[Key(0)]publicintFoo{get;set;}[Key(1)]publicintBar{get;set;}}[MessagePackObject]publicclassSample2{[Key("foo")]publicintFoo{get;set;}[Key("bar")]publicintBar{get;set;}}[MessagePackObject(keyAsPropertyName:true)]publicclassSample3{// No need for a Key attributepublicintFoo{get;set;}// If want to ignore a public member, you can use the IgnoreMember attribute[IgnoreMember]publicintBar{get;set;}}// [10,20]Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample1{Foo=10,Bar=20}));// {"foo":10,"bar":20}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample2{Foo=10,Bar=20}));// {"Foo":10}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample3{Foo=10,Bar=20}));
All public instance members (fields as well as properties) will be serialized. If you want to ignore certain public members, annotate the member with a[IgnoreMember]
attribute.
Please note that any serializable struct or class must have public accessibility; private and internal structs and classes cannot be serialized!The default of requiringMessagePackObject
annotations is meant to enforce explicitness and therefore may help write more robust code.
Should you use an indexed (int
) key or a string key?We recommend using indexed keys for faster serialization and a more compact binary representation than string keys.However, the additional information in the strings of string keys can be quite useful when debugging.
When classes change or are extended, be careful about versioning.MessagePackSerializer
will initialize members to theirdefault
value if a key does not exist in the serialized binary blob, meaning members using reference types can be initialized tonull
.If you use indexed (int
) keys, the keys should start at 0 and should be sequential. If a later version stops using certain members, you should keep the obsolete members (C# provides anObsolete
attribute to annotate such members) until all other clients had a chance to update and remove their uses of these members as well. Also, when the values of indexed keys "jump" a lot, leaving gaps in the sequence, it will negatively affect the binary size, asnull
placeholders will be inserted into the resulting arrays. However, you shouldn't reuse indexes of removed members to avoid compatibility issues between clients or when trying to deserialize legacy blobs.
Example of index gaps and resulting placeholders:
[MessagePackObject]publicclassIntKeySample{[Key(3)]publicintA{get;set;}[Key(10)]publicintB{get;set;}}// [null,null,null,0,null,null,null,null,null,null,0]Console.WriteLine(MessagePackSerializer.SerializeToJson(newIntKeySample()));
If you do not want to explicitly annotate with theMessagePackObject
/Key
attributes and instead want to use MessagePack for C# more like e.g.Json.NET, you can make use of the contractless resolver.
publicclassContractlessSample{publicintMyProperty1{get;set;}publicintMyProperty2{get;set;}}vardata=newContractlessSample{MyProperty1=99,MyProperty2=9999};varbin=MessagePackSerializer.Serialize(data,MessagePack.Resolvers.ContractlessStandardResolver.Options);// {"MyProperty1":99,"MyProperty2":9999}Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// You can also set ContractlessStandardResolver as the default.// (Global state; Not recommended when writing library code)MessagePackSerializer.DefaultOptions=MessagePack.Resolvers.ContractlessStandardResolver.Options;// Now serializable...varbin2=MessagePackSerializer.Serialize(data);
If you want to serialize private members as well, you can use one of the*AllowPrivate
resolvers.
[MessagePackObject]publicclassPrivateSample{[Key(0)]intx;publicvoidSetX(intv){x=v;}publicintGetX(){returnx;}}vardata=newPrivateSample();data.SetX(9999);// You can choose either StandardResolverAllowPrivate// or ContractlessStandardResolverAllowPrivatevarbin=MessagePackSerializer.Serialize(data,MessagePack.Resolvers.DynamicObjectResolverAllowPrivate.Options);
If you want to use MessagePack for C# more like a BinaryFormatter with a typeless serialization API, use the typeless resolver and helpers. Please consult theTypeless section.
Resolvers are the way to add specialized support for custom types to MessagePack for C#. Please refer to theExtension point section.
You can use[DataContract]
annotations instead of[MessagePackObject]
ones. If type is annotated withDataContract
, you can use[DataMember]
annotations instead of[Key]
ones and[IgnoreDataMember]
instead of[IgnoreMember]
.
Then[DataMember(Order = int)]
will behave the same as[Key(int)]
,[DataMember(Name = string)]
the same as[Key(string)]
, and[DataMember]
the same as[Key(nameof(member name)]
.
UsingDataContract
, e.g. in shared libraries, makes your classes/structs independent from MessagePack for C# serialization.However, it is not supported by the analyzers nor source generator.Also, features likeUnionAttribute
,MessagePackFormatter
,SerializationConstructor
, etc can not be used.Due to this, we recommend that you use the specific MessagePack for C# annotations when possible.
MessagePack for C# supports serialization of readonly/immutable objects/members. For example, this struct can be serialized and deserialized.
[MessagePackObject]publicstructPoint{[Key(0)]publicreadonlyintX;[Key(1)]publicreadonlyintY;publicPoint(intx,inty){this.X=x;this.Y=y;}}vardata=newPoint(99,9999);varbin=MessagePackSerializer.Serialize(data);// Okay to deserialize immutable objectvarpoint=MessagePackSerializer.Deserialize<Point>(bin);
MessagePackSerializer
will choose the constructor with the best matched argument list, using argument indexes index for index keys, or parameter names for string keys. If it cannot determine an appropriate constructor, aMessagePackDynamicObjectResolverException: can't find matched constructor parameter
exception will be thrown.You can specify which constructor to use manually with a[SerializationConstructor]
annotation.
[MessagePackObject]publicstructPoint{[Key(0)]publicreadonlyintX;[Key(1)]publicreadonlyintY;[SerializationConstructor]publicPoint(intx){this.X=x;this.Y=-1;}// If not marked attribute, used this(most matched argument)publicPoint(intx,inty){this.X=x;this.Y=y;}}
C# 9.0 record with primary constructor is similar immutable object, also supports serialize/deserialize.
// use key as property name[MessagePackObject(true)]publicrecordPoint(intX,intY);// use property: to set KeyAttribute[MessagePackObject]publicrecordPoint([property:Key(0)]intX,[property:Key(1)]intY);// Or use explicit properties[MessagePackObject]publicrecordPerson{[Key(0)]publicstringFirstName{get;init;}[Key(1)]publicstringLastName{get;init;}}
When usinginit
property setters ingeneric classes,a CLR bug prevents our most efficient code generation from invoking the property setter.As a result, you should avoid usinginit
on property setters in generic classes when using the public-onlyDynamicObjectResolver
/StandardResolver
.
When using theDynamicObjectResolverAllowPrivate
/StandardResolverAllowPrivate
resolver the bug does not apply and you may useinit
without restriction.
Objects implementing theIMessagePackSerializationCallbackReceiver
interface will receivedOnBeforeSerialize
andOnAfterDeserialize
calls during serialization/deserialization.
[MessagePackObject]publicclassSampleCallback:IMessagePackSerializationCallbackReceiver{[Key(0)]publicintKey{get;set;}publicvoidOnBeforeSerialize(){Console.WriteLine("OnBefore");}publicvoidOnAfterDeserialize(){Console.WriteLine("OnAfter");}}
MessagePack for C# supports serializing interface-typed and abstract class-typed objects. It behaves likeXmlInclude
orProtoInclude
. In MessagePack for C# these are calledUnion
. Only interfaces and abstracts classes are allowed to be annotated withUnion
attributes. Unique union keys are required.
// Annotate inheritance types[MessagePack.Union(0,typeof(FooClass))][MessagePack.Union(1,typeof(BarClass))]publicinterfaceIUnionSample{}[MessagePackObject]publicclassFooClass:IUnionSample{[Key(0)]publicintXYZ{get;set;}}[MessagePackObject]publicclassBarClass:IUnionSample{[Key(0)]publicstringOPQ{get;set;}}// ---IUnionSampledata=newFooClass(){XYZ=999};// Serialize interface-typed object.varbin=MessagePackSerializer.Serialize(data);// Deserialize again.varreData=MessagePackSerializer.Deserialize<IUnionSample>(bin);// Use with e.g. type-switching in C# 7.0switch(reData){caseFooClassx:Console.WriteLine(x.XYZ);break;caseBarClassx:Console.WriteLine(x.OPQ);break;default:break;}
Unions are internally serialized to two-element arrays.
IUnionSampledata=newBarClass{OPQ="FooBar"};varbin=MessagePackSerializer.Serialize(data);// Union is serialized to two-length array, [key, object]// [1,["FooBar"]]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));
UsingUnion
with abstract classes works the same way.
[Union(0,typeof(SubUnionType1))][Union(1,typeof(SubUnionType2))][MessagePackObject]publicabstractclassParentUnionType{[Key(0)]publicintMyProperty{get;set;}}[MessagePackObject]publicclassSubUnionType1:ParentUnionType{[Key(1)]publicintMyProperty1{get;set;}}[MessagePackObject]publicclassSubUnionType2:ParentUnionType{[Key(1)]publicintMyProperty2{get;set;}}
Please be mindful that you cannot reuse the same keys in derived types that are already present in the parent type, as internally a single flat array or map will be used and thus cannot have duplicate indexes/keys.
When callingMessagePackSerializer.Deserialize<object>
orMessagePackSerializer.Deserialize<dynamic>
, any values present in the blob will be converted to primitive values, i.e.bool
,char
,sbyte
,byte
,short
,int
,long
,ushort
,uint
,ulong
,float
,double
,DateTime
,string
,byte[]
,object[]
,IDictionary<object, object>
.
// Sample blob.varmodel=newDynamicModel{Name="foobar",Items=new[]{1,10,100,1000}};varblob=MessagePackSerializer.Serialize(model,ContractlessStandardResolver.Options);// Dynamic ("untyped")vardynamicModel=MessagePackSerializer.Deserialize<dynamic>(blob,ContractlessStandardResolver.Options);// You can access the data using array/dictionary indexers, as shown aboveConsole.WriteLine(dynamicModel["Name"]);// foobarConsole.WriteLine(dynamicModel["Items"][2]);// 100
Exploring object trees using the dictionary indexer syntax is the fastest option for untyped deserialization, but it is tedious to read and write.Where performance is not as important as code readability, consider deserializing withExpandoObject.
StandardResolver
andContractlessStandardResolver
can serializeobject
/anonymous typed objects.
varobjects=newobject[]{1,"aaa",newObjectFieldType{Anything=9999}};varbin=MessagePackSerializer.Serialize(objects);// [1,"aaa",[9999]]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// Support anonymous Type SerializevaranonType=new{Foo=100,Bar="foobar"};varbin2=MessagePackSerializer.Serialize(anonType,MessagePack.Resolvers.ContractlessStandardResolver.Options);// {"Foo":100,"Bar":"foobar"}Console.WriteLine(MessagePackSerializer.ConvertToJson(bin2));
Unity supports is limited.
When deserializing, the behavior will be the same as Dynamic (Untyped) Deserialization.
The typeless API is similar toBinaryFormatter
, as it will embed type information into the blobs, so no types need to be specified explicitly when calling the API.
objectmc=newSandbox.MyClass(){Age=10,FirstName="hoge",LastName="huga"};// Serialize with the typeless APIvarblob=MessagePackSerializer.Typeless.Serialize(mc);// Blob has embedded type-assembly information.// ["Sandbox.MyClass, Sandbox",10,"hoge","huga"]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// You can deserialize to MyClass again with the typeless API// Note that no type has to be specified explicitly in the Deserialize call// as type information is embedded in the binary blobvarobjModel=MessagePackSerializer.Typeless.Deserialize(bin)asMyClass;
Type information is represented by the MessagePackext
format, type code100
.
MessagePackSerializer.Typeless
is a shortcut ofSerialize/Deserialize<object>(TypelessContractlessStandardResolver.Instance)
.If you want to configure it as the default resolver, you can useMessagePackSerializer.Typeless.RegisterDefaultResolver
.
TypelessFormatter
can used standalone or combined with other resolvers.
// Replaced `object` uses the typeless resolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(new[]{MessagePack.Formatters.TypelessFormatter.Instance},new[]{MessagePack.Resolvers.StandardResolver.Instance});publicclassFoo{// use Typeless(this field only)[MessagePackFormatter(typeof(TypelessFormatter))]publicobjectBar;}
If a type's name is changed later, you can no longer deserialize old blobs. But you can specify a fallback name in such cases, providing aTypelessFormatter.BindToType
function of your own.
MessagePack.Formatters.TypelessFormatter.BindToType= typeName=>{if(typeName.StartsWith("SomeNamespace")){typeName=typeName.Replace("SomeNamespace","AnotherNamespace");}returnType.GetType(typeName,false);};
Deserializing data from an untrusted source can introduce security vulnerabilities in your application.Depending on the settings used during deserialization,untrusted data may be able to execute arbitrary code or cause a denial of service attack.Untrusted data might come from over the network from an untrusted source (e.g. any and every networked client) or can be tampered with by an intermediary when transmitted over an unauthenticated connection, or from a local storage that might have been tampered with, or many other sources. MessagePack for C# does not provide any means to authenticate data or make it tamper-resistant. Please use an appropriate method of authenticating data before deserialization - such as aMAC
.
Please be very mindful of these attack scenarios; many projects and companies, and serialization library users in general, have been bitten by untrusted user data deserialization in the past.
When deserializing untrusted data, put MessagePack into a more secure mode by configuring yourMessagePackSerializerOptions.Security
property:
varoptions=MessagePackSerializerOptions.Standard.WithSecurity(MessagePackSecurity.UntrustedData);// Pass the options explicitly for the greatest control.Tobject=MessagePackSerializer.Deserialize<T>(data,options);// Or set the security level as the default.MessagePackSerializer.DefaultOptions=options;
You should also avoid the Typeless serializer/formatters/resolvers for untrusted data as that opens the door for the untrusted data to potentially deserialize unanticipated types that can compromise security.
TheUntrustedData
mode merely hardens against some common attacks, but is no fully secure solution in itself.
Benchmarks comparing MessagePack For C# to other serializers were run onWindows 10 Pro x64 Intel Core i7-6700K 4.00GHz, 32GB RAM
. Benchmark code isavailable here - and theirversion info.ZeroFormatter andFlatBuffers have infinitely fast deserializers, so ignore their deserialization performance.
MessagePack for C# uses many techniques to improve performance.
- The serializer uses
IBufferWriter<byte>
rather thanSystem.IO.Stream
to reduce memory overhead. - Buffers are rented from pools to reduce allocations, keeping throughput high through reduced GC pressure.
- Don't create intermediate utility instances (
*Writer/*Reader
,*Context
, etc...) - Utilize dynamic code generation and JIT to avoid boxing value types. Use AOT generation on platforms that prohibit JITs.
- Cached generated formatters on static generic fields (don't use dictionary-cache because dictionary lookup is overhead). SeeResolvers
- Heavily tuned dynamic IL code generation and JIT to avoid boxing value types. SeeDynamicObjectTypeBuilder. Use AOT generation on platforms that prohibit JIT.
- Call the Primitive API directly when IL code generation determines target types to be primitive.
- Reduce branching of variable length formats when IL code generation knows the target type (integer/string) ranges
- Don't use the
IEnumerable<T>
abstraction to iterate over collections when possible,see: CollectionFormatterBase and derived collection formatters - Use pre-generated lookup tables to reduce checks of mgpack type constraints,see: MessagePackBinary
- Uses optimized type key dictionary for non-generic methods,see: ThreadsafeTypeKeyHashTable
- Avoid string key decoding for lookup maps (string key and use automata based name lookup with inlined IL code generation, see:AutomataDictionary
- To encode string keys, use pre-generated member name bytes and fixed sized byte array copies in IL, see:UnsafeMemory.cs
Before creating this library, I implemented a fast serializer withZeroFormatter#Performance. This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).
Performance varies depending on the options used. This is a micro benchmark withBenchmarkDotNet. The target object has 9 members (MyProperty1
~MyProperty9
), values are zero.
Method | Mean | Error | Scaled | Gen 0 | Allocated |
---|---|---|---|---|---|
M IntKey | 72.67 ns | NA | 1.00 | 0.0132 | 56 B |
M StringKey | 217.95 ns | NA | 3.00 | 0.0131 | 56 B |
M Typeless_IntKey | 176.71 ns | NA | 2.43 | 0.0131 | 56 B |
M Typeless_StringKey | 378.64 ns | NA | 5.21 | 0.0129 | 56 B |
MsgPackCliMap | 1,355.26 ns | NA | 18.65 | 0.1431 | 608 B |
MsgPackCliArray | 455.28 ns | NA | 6.26 | 0.0415 | 176 B |
ProtobufNet | 265.85 ns | NA | 3.66 | 0.0319 | 136 B |
Hyperion | 366.47 ns | NA | 5.04 | 0.0949 | 400 B |
JsonNetString | 2,783.39 ns | NA | 38.30 | 0.6790 | 2864 B |
JsonNetStreamReader | 3,297.90 ns | NA | 45.38 | 1.4267 | 6000 B |
JilString | 553.65 ns | NA | 7.62 | 0.0362 | 152 B |
JilStreamReader | 1,408.46 ns | NA | 19.38 | 0.8450 | 3552 B |
IntKey
,StringKey
,Typeless_IntKey
,Typeless_StringKey
are MessagePack for C# options. All MessagePack for C# options achieve zero memory allocations in the deserialization process.JsonNetString
/JilString
is deserialized from strings.JsonNetStreamReader
/JilStreamReader
is deserialized from UTF-8 byte arrays usingStreamReader
. Deserialization is normally read from Stream. Thus, it will be restored from byte arrays (or Stream) instead of strings.
MessagePack for C#IntKey
is the fastest.StringKey
is slower thanIntKey
because matching the character string of property names is required.IntKey
works by reading the array length, thenfor (array length) { binary decode }
.StringKey
works by reading map length,for (map length) { decode key, lookup key, binary decode }
, so it requires an additional two steps (decoding of keys and lookups of keys).
String key is often a useful, contractless, simple replacement of JSON, interoperability with other languages, and more robust versioning. MessagePack for C# is also optimized for string keys as much a possible. First of all, it does not decode UTF-8 byte arrays to full string for matching with the member name; instead it will look up the byte arrays as it is (to avoid decoding costs and extra memory allocations).
And It will try to match eachlong type
(per 8 character, if it is not enough, pad with 0) usingautomata and inline it when generating IL code.
This also avoids calculating the hash code of byte arrays, and the comparison can be made several times faster using the long type.
This is the sample of decompiled generated deserializer code, decompiled usingILSpy.
If the number of nodes is large, searches will use an embedded binary search.
Extra note, this is serialization benchmark result.
Method | Mean | Error | Scaled | Gen 0 | Allocated |
---|---|---|---|---|---|
IntKey | 84.11 ns | NA | 1.00 | 0.0094 | 40 B |
StringKey | 126.75 ns | NA | 1.51 | 0.0341 | 144 B |
Typeless_IntKey | 183.31 ns | NA | 2.18 | 0.0265 | 112 B |
Typeless_StringKey | 193.95 ns | NA | 2.31 | 0.0513 | 216 B |
MsgPackCliMap | 967.68 ns | NA | 11.51 | 0.1297 | 552 B |
MsgPackCliArray | 284.20 ns | NA | 3.38 | 0.1006 | 424 B |
ProtobufNet | 176.43 ns | NA | 2.10 | 0.0665 | 280 B |
Hyperion | 280.14 ns | NA | 3.33 | 0.1674 | 704 B |
ZeroFormatter | 149.95 ns | NA | 1.78 | 0.1009 | 424 B |
JsonNetString | 1,432.55 ns | NA | 17.03 | 0.4616 | 1944 B |
JsonNetStreamWriter | 1,775.72 ns | NA | 21.11 | 1.5526 | 6522 B |
JilString | 547.51 ns | NA | 6.51 | 0.3481 | 1464 B |
JilStreamWriter | 778.78 ns | NA | 9.26 | 1.4448 | 6066 B |
Of course,IntKey
is fastest butStringKey
also performs reasonably well.
The msgpack format does not provide for reusing strings in the data stream.This naturally leads the deserializer to create a newstring
object for every string encountered,even if it is equal to another string previously encountered.
When deserializing data that may contain the same strings repeatedly it can be worthwhileto have the deserializer take a little extra time to check whether it has seen a given string beforeand reuse it if it has.
To enable string interning onall string values, use a resolver that specifiesStringInterningFormatter
before any of the standard ones, like this:
varoptions=MessagePackSerializerOptions.Standard.WithResolver(CompositeResolver.Create(newIMessagePackFormatter[]{newStringInterningFormatter()},newIFormatterResolver[]{StandardResolver.Instance}));MessagePackSerializer.Deserialize<ClassOfStrings>(data,options);
If you know which fields of a particular type are likely to contain duplicate strings,you can apply the string interning formatter to just those fields so the deserializer only paysfor the interned string check where it matters most.Note that this technique requires a[MessagePackObject]
or[DataContract]
class.
[MessagePackObject]publicclassClassOfStrings{[Key(0)][MessagePackFormatter(typeof(StringInterningFormatter))]publicstringInternedString{get;set;}[Key(1)]publicstringOrdinaryString{get;set;}}
If you are writing your own formatter for some type that contains strings,you can call on theStringInterningFormatter
directly from your formatter as well for the strings.
MessagePack is a fast andcompact format but it is not compression.LZ4 is an extremely fast compression algorithm, and using it MessagePack for C# can achieve extremely fast performance as well as extremely compact binary sizes!
MessagePack for C# has built-in LZ4 support. You can activate it using a modified options object and passing it into an API like this:
varlz4Options=MessagePackSerializerOptions.Standard.WithCompression(MessagePackCompression.Lz4BlockArray);MessagePackSerializer.Serialize(obj,lz4Options);
MessagePackCompression
has two modes,Lz4Block
andLz4BlockArray
. Neither is a simple binary LZ4 compression, but a special compression integrated into the serialization pipeline, using MessagePackext
code (Lz4BlockArray (98)
orLz4Block (99)
). Therefore, it is not readily compatible with compression offered in other languages.
Lz4Block
compresses an entire MessagePack sequence as a single LZ4 block. This is the simple compression that achieves best compression ratio, at the cost of copying the entire sequence when necessary to get contiguous memory.
Lz4BlockArray
compresses an entire MessagePack sequence as a array of LZ4 blocks. Compressed/decompressed blocks are chunked and thus do not enter the GC's Large-Object-Heap, but the compression ratio is slightly worse.
We recommend to useLz4BlockArray
as the default when using compression.For compatibility with MessagePack v1.x, useLz4Block
.
Regardless of which LZ4 option is set at the deserialization, both methods can be deserialized. For example, when theLz4BlockArray
option was used, binary data using eitherLz4Block
andLz4BlockArray
can be deserialized. Neither can be decompressed and hence deserialized when the compression option is set toNone
.
LZ4 compression support is using Milosz Krajewski'slz4net code with some modifications.
protobuf-net is major, widely used binary-format library on .NET. I love protobuf-net and respect their great work. But when you use protobuf-net as a general purpose serialization format, you may encounter an annoying issue.
[ProtoContract]publicclassParent{[ProtoMember(1)]publicintPrimitive{get;set;}[ProtoMember(2)]publicChildProp{get;set;}[ProtoMember(3)]publicint[]Array{get;set;}}[ProtoContract]publicclassChild{[ProtoMember(1)]publicintNumber{get;set;}}using(varms=newMemoryStream()){// serialize null.ProtoBuf.Serializer.Serialize<Parent>(ms,null);ms.Position=0;varresult=ProtoBuf.Serializer.Deserialize<Parent>(ms);Console.WriteLine(result!=null);// True, not null. but all property are zero formatted.Console.WriteLine(result.Primitive);// 0Console.WriteLine(result.Prop);// nullConsole.WriteLine(result.Array);// null}using(varms=newMemoryStream()){// serialize empty array.ProtoBuf.Serializer.Serialize<Parent>(ms,newParent{Array=System.Array.Empty<int>()});ms.Position=0;varresult=ProtoBuf.Serializer.Deserialize<Parent>(ms);Console.WriteLine(result.Array==null);// True, null!}
protobuf(-net) cannot handle null and empty collection correctly, because protobuf has nonull
representation (seethis SO answer from a protobuf-net author).
MessagePack's type system can correctly serialize the entire C# type system. This is a strong reason to recommend MessagePack over protobuf.
Protocol Buffers have good IDL andgRPC support. If you want to use IDL, I recommendGoogle.Protobuf over MessagePack.
JSON is good general-purpose format. It is simple, human-readable and thoroughly-enough specified.Utf8Json - which I created as well - adopts same architecture as MessagePack for C# and avoids encoding/decoding costs as much as possible just like this library does. If you want to know more about binary vs text formats, seeUtf8Json/which serializer should be used.
ZeroFormatter is similar asFlatBuffers but specialized to C#, and special in that regard. Deserialization is infinitely fast but the produced binary size is larger. And ZeroFormatter's caching algorithm requires additional memory.
For many common uses, MessagePack for C# would be a better fit.
MessagePack for C# prioritizes maximum performance by default. However, there are also some options that sacrifice performance for convenience.
TheDeserialization Performance for different options section shows the results of indexed keys (IntKey
) vs string keys (StringKey
) performance. Indexed keys serialize the object graph as a MessagePack array. String keys serializes the object graph as a MessagePack map.
For example this type is serialized to
[MessagePackObject]publicclassPerson{[Key(0)]or[Key("name")]public string Name{get;set;}[Key(1)]or[Key("age")]public int Age{get;set;}}newPerson{Name="foobar",Age=999}
IntKey
:["foobar", 999]
StringKey
:{"name:"foobar","age":999}
.
IntKey
is always fast in both serialization and deserialization because it does not have to handle and lookup key names, and always has the smaller binary size.
StringKey
is often a useful, contractless, simple replacement for JSON, interoperability with other languages with MessagePack support, and less error prone versioning. But to achieve maximum performance, useIntKey
.
CompositeResolver.Create
is an easy way to create composite resolvers. But formatter lookups have some overhead. If you create a custom resolver (or useStaticCompositeResolver.Instance
), you can avoid this overhead.
publicclassMyApplicationResolver:IFormatterResolver{publicstaticreadonlyIFormatterResolverInstance=newMyApplicationResolver();// configure your custom resolvers.privatestaticreadonlyIFormatterResolver[]Resolvers=newIFormatterResolver[]{};privateMyApplicationResolver(){}publicIMessagePackFormatter<T>GetFormatter<T>(){returnCache<T>.Formatter;}privatestaticclassCache<T>{publicstaticIMessagePackFormatter<T>Formatter;staticCache(){// configure your custom formatters.if(typeof(T)==typeof(XXX)){Formatter=newICustomFormatter();return;}foreach(varresolverinResolvers){varf=resolver.GetFormatter<T>();if(f!=null){Formatter=f;return;}}}}}
NOTE: If you are creating a library, recommend using the above custom resolver instead of
CompositeResolver.Create
. Also, libraries must not useStaticCompositeResolver
- as it is global state - to avoid compatibility issues.
By default, MessagePack for C# serializes GUID as string. This is much slower than the native .NET format GUID. The same applies to Decimal. If your application makes heavy use of GUID or Decimal and you don't have to worry about interoperability with other languages, you can replace them with the native serializersNativeGuidResolver
andNativeDecimalResolver
respectively.
Also,DateTime
is serialized using the MessagePack timestamp format. By using theNativeDateTimeResolver
, it is possible to maintain Kind and perform faster serialization.
MessagePackSerializer.Serialize
returnsbyte[]
in default. The finalbyte[]
is copied from an internal buffer pool. That is an extra cost. You can useIBufferWriter<T>
or theStream
API to write to buffers directly. If you want to use a buffer pool outside of the serializer, you should implement customIBufferWriter<byte>
or use an existing one such asSequence<T>
from theNerdbank.Streams package.
During deserialization,MessagePackSerializer.Deserialize(ReadOnlyMemory<byte> buffer)
is better than theDeserialize(Stream stream)
overload. This is because the Stream API version starts by reading the data, generating aReadOnlySequence<byte>
, and only then starts the deserialization.
Compression is generally effective when there is duplicate data. In MessagePack, arrays containing objects using string keys (Contractless) can be compressed efficiently because compression can be applied to many duplicate property names. Indexed keys compression is not as effectively compressed as string keys, but indexed keys are smaller in the first place.
This is some example benchmark performance data;
Serializer | Mean | DataSize |
---|---|---|
IntKey | 2.941 us | 469.00 B |
IntKey(Lz4) | 3.449 us | 451.00 B |
StringKey | 4.340 us | 1023.00 B |
StringKey(Lz4) | 5.469 us | 868.00 B |
IntKey(Lz4)
is not as effectively compressed, but performance is still somewhat degraded. On the other hand,StringKey
can be expected to have a sufficient effect on the binary size. However, this is just an example. Compression can be quite effective depending on the data, too, or have little effect other than slowing down your program. There are also cases in which well-compressible data exists in the values (such as long strings, e.g. containing HTML data with many repeated HTML tags). It is important to verify the actual effects of compression on a case by case basis.
MessagePack for C# has extension points that enable you to provide optimal serialization support for custom types. There are official extension support packages.
Install-Package MessagePack.ReactivePropertyInstall-Package MessagePack.UnityShimsInstall-Package MessagePack.AspNetCoreMvcFormatter
TheMessagePack.ReactiveProperty
package adds support for types of theReactiveProperty library. It addsReactiveProperty<>
,IReactiveProperty<>
,IReadOnlyReactiveProperty<>
,ReactiveCollection<>
,Unit
serialization support. It is useful for save viewmodel state.
TheMessagePack.UnityShims
package provides shims forUnity's standard structs (Vector2
,Vector3
,Vector4
,Quaternion
,Color
,Bounds
,Rect
,AnimationCurve
,Keyframe
,Matrix4x4
,Gradient
,Color32
,RectOffset
,LayerMask
,Vector2Int
,Vector3Int
,RangeInt
,RectInt
,BoundsInt
) and corresponding formatters. It can enable proper communication between servers and Unity clients.
After installation, extension packages must be enabled, by creating composite resolvers. Here is an example showing how to enable all extensions.
// Set extensions to default resolver.varresolver=MessagePack.Resolvers.CompositeResolver.Create(// enable extension packages firstReactivePropertyResolver.Instance,MessagePack.Unity.Extension.UnityBlitResolver.Instance,MessagePack.Unity.UnityResolver.Instance,// finally use standard (default) resolverStandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);// Pass options every time or set as defaultMessagePackSerializer.DefaultOptions=options;
For configuration details, see:Extension Point section.
TheMessagePack.AspNetCoreMvcFormatter
is add-on forASP.NET Core MVC's serialization to boost up performance. This is configuration sample.
publicvoidConfigureServices(IServiceCollectionservices){services.AddMvc().AddMvcOptions(option=>{option.OutputFormatters.Clear();option.OutputFormatters.Add(newMessagePackOutputFormatter(ContractlessStandardResolver.Options));option.InputFormatters.Clear();option.InputFormatters.Add(newMessagePackInputFormatter(ContractlessStandardResolver.Options));});}
Other authors are creating extension packages, too.
- MagicOnion - gRPC based HTTP/2 RPC Streaming Framework
- MasterMemory - Embedded Readonly In-Memory Document Database
You can make your own extension serializers or integrate with frameworks. Let's create and share!
- MessagePack.FSharpExtensions - supports F# list, set, map, unit, option, discriminated union
- MessagePack.NodaTime - Support for NodaTime types to MessagePack C#
- WebApiContrib.Core.Formatter.MessagePack - supports ASP.NET Core MVC (details in blog post)
- MessagePack.MediaTypeFormatter - MessagePack MediaTypeFormatter
MessagePack for C# has experimental features which provides you with very performant formatters. There is an official package.
Install-Package MessagePack.Experimental
For detailed information, see:Experimental.md
TheMessagePackSerializer
class is the entry point of MessagePack for C#. Static methods make up the main API of MessagePack for C#.
API | Description |
---|---|
Serialize<T> | Serializes an object graph to a MessagePack binary blob. Async variant for Stream available. Non-generic overloads available. |
Deserialize<T> | Deserializes a MessagePack binary to an object graph. Async variant for Stream available. Non-generic overloads available. |
SerializeToJson | Serialize a MessagePack-compatible object graph to JSON instead of MessagePack. Useful for debugging. |
ConvertToJson | Convert MessagePack binary to JSON. Useful for debugging. |
ConvertFromJson | Convert JSON to a MessagePack binary. |
TheMessagePackSerializer.Typeless
class offers most of the same APIs as above, but removes all type arguments from the API, forcing serialization to include the full type name of the root object. It uses theTypelessContractlessStandardResolver
. Consider the result to be a .NET-specific MessagePack binary that isn't readily compatible with MessagePack deserializers in other runtimes.
MessagePack for C# fundamentally serializes usingIBufferWriter<byte>
and deserializes usingReadOnlySequence<byte>
orMemory<byte>
. Method overloads are provided to conveniently use it with common buffer types and the .NETStream
class, but some of these convenience overloads require copying buffers once and therefore have a certain overhead.
The high-level API uses a memory pool internally to avoid unnecessary memory allocation. If result size is under 64K, it allocates GC memory only for the return bytes.
Each serialize/deserialize method takes an optionalMessagePackSerializerOptions
parameter which can be used to specify a customIFormatterResolver
to use or to activate LZ4 compression support.
To deserialize aStream
that contains multiple consecutive MessagePack data structures,you can use theMessagePackStreamReader
class to efficiently identify theReadOnlySequence<byte>
for each data structure and deserialize it. For example:
staticasyncTask<List<T>>DeserializeListFromStreamAsync<T>(Streamstream,CancellationTokencancellationToken){vardataStructures=newList<T>();using(varstreamReader=newMessagePackStreamReader(stream)){while(awaitstreamReader.ReadAsync(cancellationToken)isReadOnlySequence<byte>msgpack){dataStructures.Add(MessagePackSerializer.Deserialize<T>(msgpack,cancellationToken:cancellationToken));}}returndataStructures;}
TheIMessagePackFormatter<T>
interface is responsible for serializing a unique type. For exampleInt32Formatter : IMessagePackFormatter<Int32>
represents Int32 MessagePack serializer.
publicinterfaceIMessagePackFormatter<T>{voidSerialize(refMessagePackWriterwriter,Tvalue,MessagePackSerializerOptionsoptions);TDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions);}
Many built-in formatters exists underMessagePack.Formatters
. Your custom types are usually automatically supported with the built-in type resolvers that generate newIMessagePackFormatter<T>
types on-the-fly using dynamic code generation. See ourAOT code generation support for platforms that do not support this.
However, some types - especially those provided by third party libraries or the runtime itself - cannot be appropriately annotated, and contractless serialization would produce inefficient or even wrong results.To take more control over the serialization of such custom types, write your ownIMessagePackFormatter<T>
implementation.Here is an example of such a custom formatter implementation. Note its use of the primitive API that is described in the next section.
/// <summary>Serializes a <see cref="FileInfo" /> by its full path as a string.</summary>publicclassFileInfoFormatter:IMessagePackFormatter<FileInfo>{publicvoidSerialize(refMessagePackWriterwriter,FileInfovalue,MessagePackSerializerOptionsoptions){if(value==null){writer.WriteNil();return;}writer.WriteString(value.FullName);}publicFileInfoDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){if(reader.TryReadNil()){returnnull;}options.Security.DepthStep(refreader);varpath=reader.ReadString();reader.Depth--;returnnewFileInfo(path);}}
TheDepthStep
andDepth--
statements provide a level of security while deserializing untrusted datathat might otherwise be able to execute a denial of service attack by sending MessagePack data that woulddeserialize into a very deep object graph leading to aStackOverflowException
that would crash the process.This pair of statements should surround the bulk of anyIMessagePackFormatter<T>.Deserialize
method.
Important: A message pack formatter mustread or write exactly one data structure.In the above example we just read/write a string. If you have more than one element to write out,you must precede it with a map or array header. You must read the entire map/array when deserializing.For example:
publicclassMySpecialObjectFormatter:IMessagePackFormatter<MySpecialObject>{publicvoidSerialize(refMessagePackWriterwriter,MySpecialObjectvalue,MessagePackSerializerOptionsoptions){if(value==null){writer.WriteNil();return;}writer.WriteArrayHeader(2);writer.WriteString(value.FullName);writer.WriteString(value.Age);}publicMySpecialObjectDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){if(reader.TryReadNil()){returnnull;}options.Security.DepthStep(refreader);stringfullName=null;intage=0;// Loop over *all* array elements independently of how many we expect,// since if we're serializing an older/newer version of this object it might// vary in number of elements that were serialized, but the contract of the formatter// is that exactly one data structure must be read, regardless.// Alternatively, we could check that the size of the array/map is what we expect// and throw if it is not.intcount=reader.ReadArrayHeader();for(inti=0;i<count;i++){switch(i){case0:fullName=reader.ReadString();break;case1:age=reader.ReadInt32();break;default:reader.Skip();break;}}reader.Depth--;returnnewMySpecialObject(fullName,age);}}
Your custom formatters must be discoverable via someIFormatterResolver
. Learn more in ourresolvers section.
You can see many other samples frombuiltin formatters.
TheMessagePackWriter
andMessagePackReader
structs make up the lowest-level API. They read and write the primitives types defined in the MessagePack specification.
AMessagePackReader
can efficiently read fromReadOnlyMemory<byte>
orReadOnlySequence<byte>
without any allocations, except to allocate a newstring
as required by theReadString()
method. All other methods return either value structs orReadOnlySequence<byte>
slices for extensions/arrays.Reading directly fromReadOnlySequence<byte>
means the reader can directly consume some modern high performance APIs such asPipeReader
.
Method | Description |
---|---|
Skip | Advances the reader's position past the current value. If the value is complex (e.g. map, array) the entire structure is skipped. |
Read* | Read and return a value whose type is named by the method name from the current reader position. Throws if the expected type does not match the actual type. When reading numbers, the type need not match the binary-specified type exactly. The numeric value will be coerced into the desired type or throw if the integer type is too small for a large value. |
TryReadNil | Advances beyond the current value if the current value isnil and returnstrue ; otherwise leaves the reader's position unchanged and returnsfalse . |
ReadBytes | Returns a slice of the input sequence representing the contents of abyte[] , and advances the reader. |
ReadStringSequence | Returns a slice of the input sequence representing the contents of astring without decoding it, and advances the reader. |
Clone | Creates a newMessagePackReader with the specified input sequence and the same settings as the original reader. |
CreatePeekReader | Creates a new reader with the same position as this one, allowing the caller to "read ahead" without impacting the original reader's position. |
NextCode | Reads the low-level MessagePackbyte that describes the type of the next value. Does not advance the reader. SeeMessagePack format of first byte. Its static class hasToMessagePackType andToFormatName utility methods.MessagePackRange means Min-Max fix range of MessagePack format. |
NextMessagePackType | Describes theNextCode value as a higher level category. Does not advance the reader. SeeMessagePack spec of source types. |
(others) | Other methods and properties as described by the .xml doc comment file and Intellisense. |
TheMessagePackReader
is capable of automatically interpreting both the old and new MessagePack spec.
AMessagePackWriter
writes to a given instance ofIBufferWriter<byte>
. Several common implementations of this exist, allowing zero allocations and minimal buffer copies while writing directly to several I/O APIs includingPipeWriter
.
TheMessagePackWriter
writes the new MessagePack spec by default, but can write MessagePack compatible with the old spec by setting theOldSpec
property totrue
.
Method | Description |
---|---|
Clone | Creates a newMessagePackWriter with the specified underlyingIBufferWriter<byte> and the same settings as the original writer. |
Flush | Writes any buffered bytes to the underlyingIBufferWriter<byte> . |
WriteNil | Writes the MessagePack equivalent of .NET'snull value. |
Write | Writes any MessagePack primitive value in the most compact form possible. Has overloads for every primitive type defined by the MessagePack spec. |
Write*IntType* | Writes an integer value in exactly the MessagePack type specified, even if a more compact format exists. |
WriteMapHeader | Introduces a map by specifying the number of key=value pairs it contains. |
WriteArrayHeader | Introduces an array by specifying the number of elements it contains. |
WriteExtensionFormat | Writes the full content of an extension value including length, type code and content. |
WriteExtensionFormatHeader | Writes just the header (length and type code) of an extension value. |
WriteRaw | Copies the specified bytes directly to the underlyingIBufferWriter<byte> without any validation. |
(others) | Other methods and properties as described by the .xml doc comment file and Intellisense. |
DateTime
is serialized toMessagePack Timestamp format, it serialize/deserialize UTC and losesKind
info and requires thatMessagePackWriter.OldSpec == false
.If you use theNativeDateTimeResolver
,DateTime
values will be serialized using .NET's nativeInt64
representation, which preservesKind
info but may not be interoperable with non-.NET platforms.
AnIFormatterResolver
is storage of typed serializers. TheMessagePackSerializer
API accepts aMessagePackSerializerOptions
object which specifies theIFormatterResolver
to use, allowing customization of the serialization of complex types.
Resolver Name | Description |
---|---|
BuiltinResolver | Builtin primitive and standard classes resolver. It includes primitive(int, bool, string...) and there nullable, array and list. and some extra builtin types(Guid ,Uri ,BigInteger , etc...). |
StandardResolver | Composited resolver. It resolves in the following orderbuiltin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamic object fallback . This is the default of MessagePackSerializer. |
ContractlessStandardResolver | CompositedStandardResolver (except dynamic object fallback) ->DynamicContractlessObjectResolver ->DynamicObjectTypeFallbackResolver . It enables contractless serialization. |
StandardResolverAllowPrivate | Same as StandardResolver but allow serialize/deserialize private members. |
ContractlessStandardResolverAllowPrivate | Same as ContractlessStandardResolver but allow serialize/deserialize private members. |
PrimitiveObjectResolver | MessagePack primitive object resolver. It is used fallback inobject type and supportsbool ,char ,sbyte ,byte ,short ,int ,long ,ushort ,uint ,ulong ,float ,double ,DateTime ,string ,byte[] ,ICollection ,IDictionary . |
DynamicObjectTypeFallbackResolver | Serialize is used type in fromobject type, deserialize is used PrimitiveObjectResolver. |
AttributeFormatterResolver | Get formatter from[MessagePackFormatter] attribute. |
CompositeResolver | Composes several resolvers and/or formatters together in an ordered list, allowing reuse and overriding of behaviors of existing resolvers and formatters. |
NativeDateTimeResolver | Serialize by .NET native DateTime binary format. It keepsDateTime.Kind that loses by standard(MessagePack timestamp) format. |
NativeGuidResolver | Serialize by .NET native Guid binary representation. It is faster than standard(string) representation. |
NativeDecimalResolver | Serialize by .NET native decimal binary representation. It is faster than standard(string) representation. |
DynamicEnumResolver | Resolver of enum and there nullable, serialize there underlying type. It uses dynamic code generation to avoid boxing and boostup performance serialize there name. |
DynamicEnumAsStringResolver | Resolver of enum and there nullable. It uses reflection call for resolve nullable at first time. |
DynamicGenericResolver | Resolver of generic type(Tuple<> ,List<> ,Dictionary<,> ,Array , etc). It uses reflection call for resolve generic argument at first time. |
DynamicUnionResolver | Resolver of interface marked by UnionAttribute. It uses dynamic code generation to create dynamic formatter. |
DynamicObjectResolver | Resolver of class and struct made by MessagePackObjectAttribute. It uses dynamic code generation to create dynamic formatter. |
DynamicContractlessObjectResolver | Resolver of all classes and structs. It does not needsMessagePackObjectAttribute and serialized key as string(same as marked[MessagePackObject(true)] ). |
DynamicObjectResolverAllowPrivate | Same as DynamicObjectResolver but allow serialize/deserialize private members. |
DynamicContractlessObjectResolverAllowPrivate | Same as DynamicContractlessObjectResolver but allow serialize/deserialize private members. |
TypelessObjectResolver | Used forobject , embed .NET type in binary byext(100) format so no need to pass type in deserialization. |
TypelessContractlessStandardResolver | Composited resolver. It resolves in the following ordernativedatetime -> builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamiccontractless -> typeless . This is the default ofMessagePackSerializer.Typeless |
Each instance ofMessagePackSerializer
accepts only a single resolver. Most object graphs will need more than one for serialization, so composing a single resolver made up of several is often required, and can be done with theCompositeResolver
as shown below:
// Do this once and store it for reuse.varresolver=MessagePack.Resolvers.CompositeResolver.Create(// resolver custom types firstReactivePropertyResolver.Instance,MessagePack.Unity.Extension.UnityBlitResolver.Instance,MessagePack.Unity.UnityResolver.Instance,// finally use standard resolverStandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);// Each time you serialize/deserialize, specify the options:byte[]msgpackBytes=MessagePackSerializer.Serialize(myObject,options);TmyObject2=MessagePackSerializer.Deserialize<MyObject>(msgpackBytes,options);
A resolver can be set as default withMessagePackSerializer.DefaultOptions = options
, butWARNING:When developing an application where you control all MessagePack-related code it may be safe to rely on this mutable static to control behavior.For all other libraries or multi-purpose applications that useMessagePackSerializer
you should explicitly specify theMessagePackSerializerOptions
to use with each method invocation to guarantee your code behaves as you expect even when sharing anAppDomain
or process with other MessagePack users that may change this static property.
Here is sample of useDynamicEnumAsStringResolver
withDynamicContractlessObjectResolver
(It is Json.NET-like lightweight setting.)
// composite same as StandardResolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(MessagePack.Resolvers.BuiltinResolver.Instance,MessagePack.Resolvers.AttributeFormatterResolver.Instance,// replace enum resolverMessagePack.Resolvers.DynamicEnumAsStringResolver.Instance,MessagePack.Resolvers.DynamicGenericResolver.Instance,MessagePack.Resolvers.DynamicUnionResolver.Instance,MessagePack.Resolvers.DynamicObjectResolver.Instance,MessagePack.Resolvers.PrimitiveObjectResolver.Instance,// final fallback(last priority)MessagePack.Resolvers.DynamicContractlessObjectResolver.Instance);
If you want to make an extension package, you should write both a formatter and resolverfor easier consumption.Here is sample of a resolver:
publicclassSampleCustomResolver:IFormatterResolver{// Resolver should be singleton.publicstaticreadonlyIFormatterResolverInstance=newSampleCustomResolver();privateSampleCustomResolver(){}// GetFormatter<T>'s get cost should be minimized so use type cache.publicIMessagePackFormatter<T>GetFormatter<T>(){returnFormatterCache<T>.Formatter;}privatestaticclassFormatterCache<T>{publicstaticreadonlyIMessagePackFormatter<T>Formatter;// generic's static constructor should be minimized for reduce type generation size!// use outer helper method.staticFormatterCache(){Formatter=(IMessagePackFormatter<T>)SampleCustomResolverGetFormatterHelper.GetFormatter(typeof(T));}}}internalstaticclassSampleCustomResolverGetFormatterHelper{// If type is concrete type, use type-formatter mapstaticreadonlyDictionary<Type,object>formatterMap=newDictionary<Type,object>(){{typeof(FileInfo),newFileInfoFormatter()}// add more your own custom serializers.};internalstaticobjectGetFormatter(Typet){objectformatter;if(formatterMap.TryGetValue(t,outformatter)){returnformatter;}// If type can not get, must return null for fallback mechanism.returnnull;}}
MessagePackFormatterAttribute is a lightweight extension point of class, struct, interface, enum and property/field. This is like Json.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter.
[MessagePackFormatter(typeof(CustomObjectFormatter))]publicclassCustomObject{stringinternalId;publicCustomObject(){this.internalId=Guid.NewGuid().ToString();}// serialize/deserialize internal field.classCustomObjectFormatter:IMessagePackFormatter<CustomObject>{publicvoidSerialize(refMessagePackWriterwriter,CustomObjectvalue,MessagePackSerializerOptionsoptions){options.Resolver.GetFormatterWithVerify<string>().Serialize(refwriter,value.internalId,options);}publicCustomObjectDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){varid=options.Resolver.GetFormatterWithVerify<string>().Deserialize(refreader,options);returnnewCustomObject{internalId=id};}}}// per field, memberpublicclassInt_x10Formatter:IMessagePackFormatter<int>{publicintDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){returnreader.ReadInt32()*10;}publicvoidSerialize(refMessagePackWriterwriter,intvalue,MessagePackSerializerOptionsoptions){writer.WriteInt32(value*10);}}[MessagePackObject]publicclassMyClass{// You can attach custom formatter per member.[Key(0)][MessagePackFormatter(typeof(Int_x10Formatter))]publicintMyProperty1{get;set;}}
Formatter is retrieved byAttributeFormatterResolver
, it is included inStandardResolver
.
IgnoreFormatter<T>
is lightweight extension point of class and struct. If there exists types that can't be serialized, you can registerIgnoreFormatter<T>
that serializes those to nil/null.
// CompositeResolver can set custom formatter.varresolver=MessagePack.Resolvers.CompositeResolver.Create(newIMessagePackFormatter[]{// for example, register reflection infos (can not serialize)newIgnoreFormatter<MethodBase>(),newIgnoreFormatter<MethodInfo>(),newIgnoreFormatter<PropertyInfo>(),newIgnoreFormatter<FieldInfo>()},newIFormatterResolver[]{ContractlessStandardResolver.Instance});
MessagePack for C# already used some MessagePack extension type codes, be careful to avoid using the same ext code for other purposes.
Range | Reserved for |
---|---|
[-128, -1] | Reserved by the msgpack spec for predefined types |
[30, 120) | Reserved for this library's use to support common types in .NET |
This leaves the following ranges for your use:
- [0, 30)
- [120, 127]
Within thereserved ranges, this library defines or implements extensions that use these type codes:
Code | Type | Use by |
---|---|---|
-1 | DateTime | MessagePack-spec reserved for timestamp |
30 | Vector2[] | for Unity, UnsafeBlitFormatter |
31 | Vector3[] | for Unity, UnsafeBlitFormatter |
32 | Vector4[] | for Unity, UnsafeBlitFormatter |
33 | Quaternion[] | for Unity, UnsafeBlitFormatter |
34 | Color[] | for Unity, UnsafeBlitFormatter |
35 | Bounds[] | for Unity, UnsafeBlitFormatter |
36 | Rect[] | for Unity, UnsafeBlitFormatter |
37 | Int[] | for Unity, UnsafeBlitFormatter |
38 | Float[] | for Unity, UnsafeBlitFormatter |
39 | Double[] | for Unity, UnsafeBlitFormatter |
98 | All | MessagePackCompression.Lz4BlockArray |
99 | All | MessagePackCompression.Lz4Block |
100 | object | TypelessFormatter |
The minimum supported Unity version will be2022.3.12f1
, as it is necessary to support IL2CPP via C# Source Generator.
There are two installation steps required to use it in Unity. Do both, not just one.
Install
MessagePack
from NuGet usingNuGetForUnityOpen Window from NuGet -> Manage NuGet Packages, Search "MessagePack" and Press Install.Install
MessagePack.Unity
package by referencing the git URL.Open Package Manager window and pressAdd Package from git URL...
, enter following pathhttps://github.com/MessagePack-CSharp/MessagePack-CSharp.git?path=src/MessagePack.UnityClient/Assets/Scripts/MessagePack
MessagePack uses the ..* release tag, so you can specify a version like #v3.0.0. For example:
https://github.com/MessagePack-CSharp/MessagePack-CSharp.git?path=src/MessagePack.UnityClient/Assets/Scripts/MessagePack#v3.0.0
In Unity, MessagePackSerializer can serializeVector2
,Vector3
,Vector4
,Quaternion
,Color
,Bounds
,Rect
,AnimationCurve
,Keyframe
,Matrix4x4
,Gradient
,Color32
,RectOffset
,LayerMask
,Vector2Int
,Vector3Int
,RangeInt
,RectInt
,BoundsInt
and their nullable, array and list types with the built-in extensionUnityResolver
.
MessagePack.Unity
automatically addsUnityResolver
to the default options Resolver when the application starts with code like this in the unity package to enable this serialization:
[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.SubsystemRegistration)]privatestaticvoidInit(){MessagePackSerializer.DefaultOptions=MessagePackSerializerOptions.Standard.WithResolver(UnityResolver.InstanceWithStandardResolver);}
If you want to customize the Resolver or change the DefaultOptions, it would be good to keep this in mind.
TheMessagePack.UnityShims
NuGet package is for .NET server-side serialization support to communicate with Unity. It includes shims for Vector3 etc and the Safe/Unsafe serialization extension.
There are several ways to share types between .NET and Unity:
- Share using symbolic links
- Place the actual files on the Unity side and reference them as link files in the .NET csproj
- Use UPM local references to reference the .NET project from the Unity side
While the setup is a bit challenging, the smoothest way to share is using UPM local references. For detailed steps, please refer to theMagicOnion Sample.
MessagePack for C# has an additional unsafe extension.UnsafeBlitResolver
is special resolver for extremely fast but unsafe serialization/deserialization of struct arrays.
x20 faster Vector3[] serialization than native JsonUtility. If useUnsafeBlitResolver
, serialization uses a special format (ext:typecode 30~39) forVector2[]
,Vector3[]
,Quaternion[]
,Color[]
,Bounds[]
,Rect[]
. If useUnityBlitWithPrimitiveArrayResolver
, it supportsint[]
,float[]
,double[]
too. This special feature is useful for serializing Mesh (manyVector3[]
) or many transform positions.
If you want to use unsafe resolver, registerUnityBlitResolver
orUnityBlitWithPrimitiveArrayResolver
.
Here is sample of configuration.
StaticCompositeResolver.Instance.Register(MessagePack.Unity.UnityResolver.Instance,MessagePack.Unity.Extension.UnityBlitWithPrimitiveArrayResolver.Instance,MessagePack.Resolvers.StandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(StaticCompositeResolver.Instance);MessagePackSerializer.DefaultOptions=options;
A source generator is provided in theMessagePackAnalyzer
package, which is automatically installed when you installMessagePack
via NuGet.This will source generate the formatters required for all your[MessagePackObject]
-annotated data types during compilation for the fastest possible startup and runtime.AnIFormatterResolver
is also generated that bundles all source generated and user-written formatters together.TheStandardResolver
includes theSourceGeneratedFormatterResolver
which discovers and uses your source generated resolver automatically.
Therefore, in the usual scenario, it will work with AOT Safe without any special handling.If you prefer to restrict your resolver to source-generated formatters, you should use theMessagePack.GeneratedMessagePackResolver
, which is source generated into your project for that purpose.This type's name and namespace can be customized by applying[GeneratedMessagePackResolver]
to apartial class
that you define, at which point that class becomes the resolver for you to use.
At runtime, if a source generated or hand-written formatter cannot be found for a given[MessagePackObject]
type, MessagePack will generate the formatters on the fly usingReflection.Emit to create highly-tuned formatters for each type.This code generation has a minor upfront performance cost.
Note: When using Unity, dynamic code generation only works when targeting .NET Framework 4.x + mono runtime.For all other Unity targets, AOT is required.
You can customize the generated source through properties on theGeneratedMessagePackResolverAttribute
.
[GeneratedMessagePackResolver]partialclassMyResolver{}
When exposing the generated resolver publicly, consumers outside the library should aggregate the resolver using itsInstance
property, which containsonly the generated formatters.
Two assembly-level attributes exist to help with mixing in your own custom formatters with the automatically generated ones:
MessagePackKnownFormatterAttribute
MessagePackAssumedFormattableAttribute
Learn more about using a mix of your own custom formatters and automatically generated ones inthe Analyzer section.
MagicOnion is a code-first gRPC framework based on grpc-dotnet and MessagePack. gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not need IDL. When communicating C# to C#, schemaless (or rather C# classes as schema) is better than using IDL.
The StreamJsonRpc library is based onJSON-RPC and includesa pluggable formatter architecture and as of v2.3 includesMessagePack support.
See ourcontributor's guide.
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community.For more information see the.NET Foundation Code of Conduct.
This project is supported by the.NET Foundation.
About
Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#]
Topics
Resources
License
Contributing
Security policy
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Sponsor this project
Uh oh!
There was an error while loading.Please reload this page.