Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#]

License

NotificationsYou must be signed in to change notification settings

MessagePack-CSharp/MessagePack-CSharp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NuGetNuGetReleases

Join the chat at https://gitter.im/MessagePack-CSharp/LobbyBuild Status

The extremely fastMessagePack serializer for C#.It is 10x faster thanMsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like games, distributed computing, microservices, or data caches.

Perf comparison graph

MessagePack has a compact binary size and a full set of general purpose expressive data types. Please have a look at thecomparison with JSON, protobuf, ZeroFormatter section and learnwhy MessagePack C# is the fastest.

Table of Contents

Installation

This library is distributed via NuGet. SpecialUnity support is available, too.

We target .NET Standard 2.0 with special optimizations for .NET 8+ and .NET Framework.The library code is pure C# (with Just-In-Time IL code generation on some platforms or AOT safe source generators).

NuGet packages

To install with NuGet, just install theMessagePack package:

Install-Package MessagePack

There are also a range of official and third party Extension Packages available (learn more in ourextensions section):

Install-Package MessagePack.ReactivePropertyInstall-Package MessagePack.UnityShimsInstall-Package MessagePack.AspNetCoreMvcFormatter

Unity

For Unity projects, please read theUnity Support section to install.

Migration notes from prior versions

Migrating from a prior major version of MessagePack to the latest?Check outthese instructions.

Quick Start

Define the struct or class to be serialized and annotate it with a[MessagePackObject] attribute.Annotate members whose values should be serialized (fields as well as properties) with[Key] attributes.

[MessagePackObject]publicclassMyClass{// Key attributes take a serialization index (or string name)// The values must be unique and versioning has to be considered as well.// Keys are described in later sections in more detail.[Key(0)]publicintAge{get;set;}[Key(1)]publicstringFirstName{get;set;}[Key(2)]publicstringLastName{get;set;}// All fields or properties that should not be serialized must be annotated with [IgnoreMember].[IgnoreMember]publicstringFullName{get{returnFirstName+LastName;}}}

CallMessagePackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize your object instance.You can use theConvertToJson method to get a human readable representation of any MessagePack binary blob.

classProgram{staticvoidMain(string[]args){varmc=newMyClass{Age=99,FirstName="hoge",LastName="huga",};// Call Serialize/Deserialize, that's all.byte[]bytes=MessagePackSerializer.Serialize(mc);MyClassmc2=MessagePackSerializer.Deserialize<MyClass>(bytes);// You can dump MessagePack binary blobs to human readable json.// Using indexed keys (as opposed to string keys) will serialize to MessagePack arrays,// hence property names are not available.// [99,"hoge","huga"]varjson=MessagePackSerializer.ConvertToJson(bytes);Console.WriteLine(json);}}

By default, aMessagePackObject annotation is required. This can be made optional; see theObject Serialization section and theFormatter Resolver section for details.

Analyzer

The MessagePackAnalyzer package(includes in default) aids with:

  1. Produces compiler warnings upon incorrect attribute use, member accessibility, and more.
  2. Automating attributing of your serializable classes and members.
  3. Optionally improving startup time throughAOT formatter generation.

The first two of these features is demonstrated below:

analyzergif

Two assembly-level attributes exist to help with mixing in your own custom formatters with the automatically generated ones:

  • MessagePackKnownFormatterAttribute - Identifies classes that implementIMessagePackFormatter<T>.TheT type argument willnot produce an analyzer warning whenT is used elsewhere in a serializable object.When using a source generated resolver, the resolver will refer to this formatter for the appropriate type(s).
  • MessagePackAssumedFormattableAttribute - Identifies types that are assumed to have anIMessagePackFormatter<T>somewhere, and that will be combined within anIFormatterResolver at runtime to ensure the specified type can be serialized.This attribute will suppress the analyzer warning from using that type although the type does not have a[MessagePackObject] attribute on it.

Built-in supported types

These types can serialize by default:

  • Primitives (int,string, etc...),Enums,Nullable<>,Lazy<>
  • TimeSpan,DateTime,DateTimeOffset
  • Guid,Uri,Version,StringBuilder
  • BigInteger,Complex,Half
  • Array[],Array[,],Array[,,],Array[,,,],ArraySegment<>,BitArray
  • KeyValuePair<,>,Tuple<,...>,ValueTuple<,...>
  • ArrayList,Hashtable
  • List<>,LinkedList<>,Queue<>,Stack<>,HashSet<>,ReadOnlyCollection<>,SortedList<,>
  • IList<>,ICollection<>,IEnumerable<>,IReadOnlyCollection<>,IReadOnlyList<>
  • Dictionary<,>,IDictionary<,>,SortedDictionary<,>,ILookup<,>,IGrouping<,>,ReadOnlyDictionary<,>,IReadOnlyDictionary<,>
  • ObservableCollection<>,ReadOnlyObservableCollection<>
  • ISet<>,
  • ConcurrentBag<>,ConcurrentQueue<>,ConcurrentStack<>,ConcurrentDictionary<,>
  • Immutable collections (ImmutableList<>, etc)
  • Custom implementations ofICollection<> orIDictionary<,> with a parameterless constructor
  • Custom implementations ofIList orIDictionary with a parameterless constructor

You can add support for custom types, and there are some official/third-party extension packages for:

  • ReactiveProperty
  • for Unity (Vector3,Quaternion, etc...)
  • F# (Record, FsList, Discriminated Unions, etc...)

Please see theextensions section.

MessagePack.Nil is the built-in type representing null/void in MessagePack for C#.

Object Serialization

MessagePack for C# can serialize your own publicclass orstruct types. By default, serializable types must be annotated with the[MessagePackObject] attribute and members with the[Key] attribute. Keys can be either indexes (int) or arbitrary strings. If all keys are indexes, arrays are used for serialization, which offers advantages in performance and binary size. Otherwise, MessagePack maps (dictionaries) will be used.

If you use[MessagePackObject(keyAsPropertyName: true)], then members do not require explicitKey attributes, but string keys will be used.

[MessagePackObject]publicclassSample1{[Key(0)]publicintFoo{get;set;}[Key(1)]publicintBar{get;set;}}[MessagePackObject]publicclassSample2{[Key("foo")]publicintFoo{get;set;}[Key("bar")]publicintBar{get;set;}}[MessagePackObject(keyAsPropertyName:true)]publicclassSample3{// No need for a Key attributepublicintFoo{get;set;}// If want to ignore a public member, you can use the  IgnoreMember attribute[IgnoreMember]publicintBar{get;set;}}// [10,20]Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample1{Foo=10,Bar=20}));// {"foo":10,"bar":20}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample2{Foo=10,Bar=20}));// {"Foo":10}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample3{Foo=10,Bar=20}));

All public instance members (fields as well as properties) will be serialized. If you want to ignore certain public members, annotate the member with a[IgnoreMember] attribute.

Please note that any serializable struct or class must have public accessibility; private and internal structs and classes cannot be serialized!The default of requiringMessagePackObject annotations is meant to enforce explicitness and therefore may help write more robust code.

Should you use an indexed (int) key or a string key?We recommend using indexed keys for faster serialization and a more compact binary representation than string keys.However, the additional information in the strings of string keys can be quite useful when debugging.

When classes change or are extended, be careful about versioning.MessagePackSerializer will initialize members to theirdefault value if a key does not exist in the serialized binary blob, meaning members using reference types can be initialized tonull.If you use indexed (int) keys, the keys should start at 0 and should be sequential. If a later version stops using certain members, you should keep the obsolete members (C# provides anObsolete attribute to annotate such members) until all other clients had a chance to update and remove their uses of these members as well. Also, when the values of indexed keys "jump" a lot, leaving gaps in the sequence, it will negatively affect the binary size, asnull placeholders will be inserted into the resulting arrays. However, you shouldn't reuse indexes of removed members to avoid compatibility issues between clients or when trying to deserialize legacy blobs.

Example of index gaps and resulting placeholders:

[MessagePackObject]publicclassIntKeySample{[Key(3)]publicintA{get;set;}[Key(10)]publicintB{get;set;}}// [null,null,null,0,null,null,null,null,null,null,0]Console.WriteLine(MessagePackSerializer.SerializeToJson(newIntKeySample()));

If you do not want to explicitly annotate with theMessagePackObject/Key attributes and instead want to use MessagePack for C# more like e.g.Json.NET, you can make use of the contractless resolver.

publicclassContractlessSample{publicintMyProperty1{get;set;}publicintMyProperty2{get;set;}}vardata=newContractlessSample{MyProperty1=99,MyProperty2=9999};varbin=MessagePackSerializer.Serialize(data,MessagePack.Resolvers.ContractlessStandardResolver.Options);// {"MyProperty1":99,"MyProperty2":9999}Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// You can also set ContractlessStandardResolver as the default.// (Global state; Not recommended when writing library code)MessagePackSerializer.DefaultOptions=MessagePack.Resolvers.ContractlessStandardResolver.Options;// Now serializable...varbin2=MessagePackSerializer.Serialize(data);

If you want to serialize private members as well, you can use one of the*AllowPrivate resolvers.

[MessagePackObject]publicclassPrivateSample{[Key(0)]intx;publicvoidSetX(intv){x=v;}publicintGetX(){returnx;}}vardata=newPrivateSample();data.SetX(9999);// You can choose either StandardResolverAllowPrivate// or ContractlessStandardResolverAllowPrivatevarbin=MessagePackSerializer.Serialize(data,MessagePack.Resolvers.DynamicObjectResolverAllowPrivate.Options);

If you want to use MessagePack for C# more like a BinaryFormatter with a typeless serialization API, use the typeless resolver and helpers. Please consult theTypeless section.

Resolvers are the way to add specialized support for custom types to MessagePack for C#. Please refer to theExtension point section.

DataContract compatibility

You can use[DataContract] annotations instead of[MessagePackObject] ones. If type is annotated withDataContract, you can use[DataMember] annotations instead of[Key] ones and[IgnoreDataMember] instead of[IgnoreMember].

Then[DataMember(Order = int)] will behave the same as[Key(int)],[DataMember(Name = string)] the same as[Key(string)], and[DataMember] the same as[Key(nameof(member name)].

UsingDataContract, e.g. in shared libraries, makes your classes/structs independent from MessagePack for C# serialization.However, it is not supported by the analyzers nor source generator.Also, features likeUnionAttribute,MessagePackFormatter,SerializationConstructor, etc can not be used.Due to this, we recommend that you use the specific MessagePack for C# annotations when possible.

Serializing readonly/immutable object members (SerializationConstructor)

MessagePack for C# supports serialization of readonly/immutable objects/members. For example, this struct can be serialized and deserialized.

[MessagePackObject]publicstructPoint{[Key(0)]publicreadonlyintX;[Key(1)]publicreadonlyintY;publicPoint(intx,inty){this.X=x;this.Y=y;}}vardata=newPoint(99,9999);varbin=MessagePackSerializer.Serialize(data);// Okay to deserialize immutable objectvarpoint=MessagePackSerializer.Deserialize<Point>(bin);

MessagePackSerializer will choose the constructor with the best matched argument list, using argument indexes index for index keys, or parameter names for string keys. If it cannot determine an appropriate constructor, aMessagePackDynamicObjectResolverException: can't find matched constructor parameter exception will be thrown.You can specify which constructor to use manually with a[SerializationConstructor] annotation.

[MessagePackObject]publicstructPoint{[Key(0)]publicreadonlyintX;[Key(1)]publicreadonlyintY;[SerializationConstructor]publicPoint(intx){this.X=x;this.Y=-1;}// If not marked attribute, used this(most matched argument)publicPoint(intx,inty){this.X=x;this.Y=y;}}

C# 9record types

C# 9.0 record with primary constructor is similar immutable object, also supports serialize/deserialize.

// use key as property name[MessagePackObject(true)]publicrecordPoint(intX,intY);// use property: to set KeyAttribute[MessagePackObject]publicrecordPoint([property:Key(0)]intX,[property:Key(1)]intY);// Or use explicit properties[MessagePackObject]publicrecordPerson{[Key(0)]publicstringFirstName{get;init;}[Key(1)]publicstringLastName{get;init;}}

C# 9init property setter limitations

When usinginit property setters ingeneric classes,a CLR bug prevents our most efficient code generation from invoking the property setter.As a result, you should avoid usinginit on property setters in generic classes when using the public-onlyDynamicObjectResolver/StandardResolver.

When using theDynamicObjectResolverAllowPrivate/StandardResolverAllowPrivate resolver the bug does not apply and you may useinit without restriction.

Serialization Callback

Objects implementing theIMessagePackSerializationCallbackReceiver interface will receivedOnBeforeSerialize andOnAfterDeserialize calls during serialization/deserialization.

[MessagePackObject]publicclassSampleCallback:IMessagePackSerializationCallbackReceiver{[Key(0)]publicintKey{get;set;}publicvoidOnBeforeSerialize(){Console.WriteLine("OnBefore");}publicvoidOnAfterDeserialize(){Console.WriteLine("OnAfter");}}

Union

MessagePack for C# supports serializing interface-typed and abstract class-typed objects. It behaves likeXmlInclude orProtoInclude. In MessagePack for C# these are calledUnion. Only interfaces and abstracts classes are allowed to be annotated withUnion attributes. Unique union keys are required.

// Annotate inheritance types[MessagePack.Union(0,typeof(FooClass))][MessagePack.Union(1,typeof(BarClass))]publicinterfaceIUnionSample{}[MessagePackObject]publicclassFooClass:IUnionSample{[Key(0)]publicintXYZ{get;set;}}[MessagePackObject]publicclassBarClass:IUnionSample{[Key(0)]publicstringOPQ{get;set;}}// ---IUnionSampledata=newFooClass(){XYZ=999};// Serialize interface-typed object.varbin=MessagePackSerializer.Serialize(data);// Deserialize again.varreData=MessagePackSerializer.Deserialize<IUnionSample>(bin);// Use with e.g. type-switching in C# 7.0switch(reData){caseFooClassx:Console.WriteLine(x.XYZ);break;caseBarClassx:Console.WriteLine(x.OPQ);break;default:break;}

Unions are internally serialized to two-element arrays.

IUnionSampledata=newBarClass{OPQ="FooBar"};varbin=MessagePackSerializer.Serialize(data);// Union is serialized to two-length array, [key, object]// [1,["FooBar"]]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));

UsingUnion with abstract classes works the same way.

[Union(0,typeof(SubUnionType1))][Union(1,typeof(SubUnionType2))][MessagePackObject]publicabstractclassParentUnionType{[Key(0)]publicintMyProperty{get;set;}}[MessagePackObject]publicclassSubUnionType1:ParentUnionType{[Key(1)]publicintMyProperty1{get;set;}}[MessagePackObject]publicclassSubUnionType2:ParentUnionType{[Key(1)]publicintMyProperty2{get;set;}}

Please be mindful that you cannot reuse the same keys in derived types that are already present in the parent type, as internally a single flat array or map will be used and thus cannot have duplicate indexes/keys.

Dynamic (Untyped) Deserialization

When callingMessagePackSerializer.Deserialize<object> orMessagePackSerializer.Deserialize<dynamic>, any values present in the blob will be converted to primitive values, i.e.bool,char,sbyte,byte,short,int,long,ushort,uint,ulong,float,double,DateTime,string,byte[],object[],IDictionary<object, object>.

// Sample blob.varmodel=newDynamicModel{Name="foobar",Items=new[]{1,10,100,1000}};varblob=MessagePackSerializer.Serialize(model,ContractlessStandardResolver.Options);// Dynamic ("untyped")vardynamicModel=MessagePackSerializer.Deserialize<dynamic>(blob,ContractlessStandardResolver.Options);// You can access the data using array/dictionary indexers, as shown aboveConsole.WriteLine(dynamicModel["Name"]);// foobarConsole.WriteLine(dynamicModel["Items"][2]);// 100

Exploring object trees using the dictionary indexer syntax is the fastest option for untyped deserialization, but it is tedious to read and write.Where performance is not as important as code readability, consider deserializing withExpandoObject.

Object Type Serialization

StandardResolver andContractlessStandardResolver can serializeobject/anonymous typed objects.

varobjects=newobject[]{1,"aaa",newObjectFieldType{Anything=9999}};varbin=MessagePackSerializer.Serialize(objects);// [1,"aaa",[9999]]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// Support anonymous Type SerializevaranonType=new{Foo=100,Bar="foobar"};varbin2=MessagePackSerializer.Serialize(anonType,MessagePack.Resolvers.ContractlessStandardResolver.Options);// {"Foo":100,"Bar":"foobar"}Console.WriteLine(MessagePackSerializer.ConvertToJson(bin2));

Unity supports is limited.

When deserializing, the behavior will be the same as Dynamic (Untyped) Deserialization.

Typeless

The typeless API is similar toBinaryFormatter, as it will embed type information into the blobs, so no types need to be specified explicitly when calling the API.

objectmc=newSandbox.MyClass(){Age=10,FirstName="hoge",LastName="huga"};// Serialize with the typeless APIvarblob=MessagePackSerializer.Typeless.Serialize(mc);// Blob has embedded type-assembly information.// ["Sandbox.MyClass, Sandbox",10,"hoge","huga"]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));// You can deserialize to MyClass again with the typeless API// Note that no type has to be specified explicitly in the Deserialize call// as type information is embedded in the binary blobvarobjModel=MessagePackSerializer.Typeless.Deserialize(bin)asMyClass;

Type information is represented by the MessagePackext format, type code100.

MessagePackSerializer.Typeless is a shortcut ofSerialize/Deserialize<object>(TypelessContractlessStandardResolver.Instance).If you want to configure it as the default resolver, you can useMessagePackSerializer.Typeless.RegisterDefaultResolver.

TypelessFormatter can used standalone or combined with other resolvers.

// Replaced `object` uses the typeless resolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(new[]{MessagePack.Formatters.TypelessFormatter.Instance},new[]{MessagePack.Resolvers.StandardResolver.Instance});publicclassFoo{// use Typeless(this field only)[MessagePackFormatter(typeof(TypelessFormatter))]publicobjectBar;}

If a type's name is changed later, you can no longer deserialize old blobs. But you can specify a fallback name in such cases, providing aTypelessFormatter.BindToType function of your own.

MessagePack.Formatters.TypelessFormatter.BindToType= typeName=>{if(typeName.StartsWith("SomeNamespace")){typeName=typeName.Replace("SomeNamespace","AnotherNamespace");}returnType.GetType(typeName,false);};

Security

Deserializing data from an untrusted source can introduce security vulnerabilities in your application.Depending on the settings used during deserialization,untrusted data may be able to execute arbitrary code or cause a denial of service attack.Untrusted data might come from over the network from an untrusted source (e.g. any and every networked client) or can be tampered with by an intermediary when transmitted over an unauthenticated connection, or from a local storage that might have been tampered with, or many other sources. MessagePack for C# does not provide any means to authenticate data or make it tamper-resistant. Please use an appropriate method of authenticating data before deserialization - such as aMAC .

Please be very mindful of these attack scenarios; many projects and companies, and serialization library users in general, have been bitten by untrusted user data deserialization in the past.

When deserializing untrusted data, put MessagePack into a more secure mode by configuring yourMessagePackSerializerOptions.Security property:

varoptions=MessagePackSerializerOptions.Standard.WithSecurity(MessagePackSecurity.UntrustedData);// Pass the options explicitly for the greatest control.Tobject=MessagePackSerializer.Deserialize<T>(data,options);// Or set the security level as the default.MessagePackSerializer.DefaultOptions=options;

You should also avoid the Typeless serializer/formatters/resolvers for untrusted data as that opens the door for the untrusted data to potentially deserialize unanticipated types that can compromise security.

TheUntrustedData mode merely hardens against some common attacks, but is no fully secure solution in itself.

Performance

Benchmarks comparing MessagePack For C# to other serializers were run onWindows 10 Pro x64 Intel Core i7-6700K 4.00GHz, 32GB RAM. Benchmark code isavailable here - and theirversion info.ZeroFormatter andFlatBuffers have infinitely fast deserializers, so ignore their deserialization performance.

image

MessagePack for C# uses many techniques to improve performance.

  • The serializer usesIBufferWriter<byte> rather thanSystem.IO.Stream to reduce memory overhead.
  • Buffers are rented from pools to reduce allocations, keeping throughput high through reduced GC pressure.
  • Don't create intermediate utility instances (*Writer/*Reader,*Context, etc...)
  • Utilize dynamic code generation and JIT to avoid boxing value types. Use AOT generation on platforms that prohibit JITs.
  • Cached generated formatters on static generic fields (don't use dictionary-cache because dictionary lookup is overhead). SeeResolvers
  • Heavily tuned dynamic IL code generation and JIT to avoid boxing value types. SeeDynamicObjectTypeBuilder. Use AOT generation on platforms that prohibit JIT.
  • Call the Primitive API directly when IL code generation determines target types to be primitive.
  • Reduce branching of variable length formats when IL code generation knows the target type (integer/string) ranges
  • Don't use theIEnumerable<T> abstraction to iterate over collections when possible,see: CollectionFormatterBase and derived collection formatters
  • Use pre-generated lookup tables to reduce checks of mgpack type constraints,see: MessagePackBinary
  • Uses optimized type key dictionary for non-generic methods,see: ThreadsafeTypeKeyHashTable
  • Avoid string key decoding for lookup maps (string key and use automata based name lookup with inlined IL code generation, see:AutomataDictionary
  • To encode string keys, use pre-generated member name bytes and fixed sized byte array copies in IL, see:UnsafeMemory.cs

Before creating this library, I implemented a fast serializer withZeroFormatter#Performance. This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).

Deserialization Performance for different options

Performance varies depending on the options used. This is a micro benchmark withBenchmarkDotNet. The target object has 9 members (MyProperty1 ~MyProperty9), values are zero.

MethodMeanErrorScaledGen 0Allocated
M IntKey72.67 nsNA1.000.013256 B
M StringKey217.95 nsNA3.000.013156 B
M Typeless_IntKey176.71 nsNA2.430.013156 B
M Typeless_StringKey378.64 nsNA5.210.012956 B
MsgPackCliMap1,355.26 nsNA18.650.1431608 B
MsgPackCliArray455.28 nsNA6.260.0415176 B
ProtobufNet265.85 nsNA3.660.0319136 B
Hyperion366.47 nsNA5.040.0949400 B
JsonNetString2,783.39 nsNA38.300.67902864 B
JsonNetStreamReader3,297.90 nsNA45.381.42676000 B
JilString553.65 nsNA7.620.0362152 B
JilStreamReader1,408.46 nsNA19.380.84503552 B

IntKey,StringKey,Typeless_IntKey,Typeless_StringKey are MessagePack for C# options. All MessagePack for C# options achieve zero memory allocations in the deserialization process.JsonNetString/JilString is deserialized from strings.JsonNetStreamReader/JilStreamReader is deserialized from UTF-8 byte arrays usingStreamReader. Deserialization is normally read from Stream. Thus, it will be restored from byte arrays (or Stream) instead of strings.

MessagePack for C#IntKey is the fastest.StringKey is slower thanIntKey because matching the character string of property names is required.IntKey works by reading the array length, thenfor (array length) { binary decode }.StringKey works by reading map length,for (map length) { decode key, lookup key, binary decode }, so it requires an additional two steps (decoding of keys and lookups of keys).

String key is often a useful, contractless, simple replacement of JSON, interoperability with other languages, and more robust versioning. MessagePack for C# is also optimized for string keys as much a possible. First of all, it does not decode UTF-8 byte arrays to full string for matching with the member name; instead it will look up the byte arrays as it is (to avoid decoding costs and extra memory allocations).

And It will try to match eachlong type (per 8 character, if it is not enough, pad with 0) usingautomata and inline it when generating IL code.

image

This also avoids calculating the hash code of byte arrays, and the comparison can be made several times faster using the long type.

This is the sample of decompiled generated deserializer code, decompiled usingILSpy.

image

If the number of nodes is large, searches will use an embedded binary search.

Extra note, this is serialization benchmark result.

MethodMeanErrorScaledGen 0Allocated
IntKey84.11 nsNA1.000.009440 B
StringKey126.75 nsNA1.510.0341144 B
Typeless_IntKey183.31 nsNA2.180.0265112 B
Typeless_StringKey193.95 nsNA2.310.0513216 B
MsgPackCliMap967.68 nsNA11.510.1297552 B
MsgPackCliArray284.20 nsNA3.380.1006424 B
ProtobufNet176.43 nsNA2.100.0665280 B
Hyperion280.14 nsNA3.330.1674704 B
ZeroFormatter149.95 nsNA1.780.1009424 B
JsonNetString1,432.55 nsNA17.030.46161944 B
JsonNetStreamWriter1,775.72 nsNA21.111.55266522 B
JilString547.51 nsNA6.510.34811464 B
JilStreamWriter778.78 nsNA9.261.44486066 B

Of course,IntKey is fastest butStringKey also performs reasonably well.

String interning

The msgpack format does not provide for reusing strings in the data stream.This naturally leads the deserializer to create a newstring object for every string encountered,even if it is equal to another string previously encountered.

When deserializing data that may contain the same strings repeatedly it can be worthwhileto have the deserializer take a little extra time to check whether it has seen a given string beforeand reuse it if it has.

To enable string interning onall string values, use a resolver that specifiesStringInterningFormatterbefore any of the standard ones, like this:

varoptions=MessagePackSerializerOptions.Standard.WithResolver(CompositeResolver.Create(newIMessagePackFormatter[]{newStringInterningFormatter()},newIFormatterResolver[]{StandardResolver.Instance}));MessagePackSerializer.Deserialize<ClassOfStrings>(data,options);

If you know which fields of a particular type are likely to contain duplicate strings,you can apply the string interning formatter to just those fields so the deserializer only paysfor the interned string check where it matters most.Note that this technique requires a[MessagePackObject] or[DataContract] class.

[MessagePackObject]publicclassClassOfStrings{[Key(0)][MessagePackFormatter(typeof(StringInterningFormatter))]publicstringInternedString{get;set;}[Key(1)]publicstringOrdinaryString{get;set;}}

If you are writing your own formatter for some type that contains strings,you can call on theStringInterningFormatter directly from your formatter as well for the strings.

LZ4 Compression

MessagePack is a fast andcompact format but it is not compression.LZ4 is an extremely fast compression algorithm, and using it MessagePack for C# can achieve extremely fast performance as well as extremely compact binary sizes!

MessagePack for C# has built-in LZ4 support. You can activate it using a modified options object and passing it into an API like this:

varlz4Options=MessagePackSerializerOptions.Standard.WithCompression(MessagePackCompression.Lz4BlockArray);MessagePackSerializer.Serialize(obj,lz4Options);

MessagePackCompression has two modes,Lz4Block andLz4BlockArray. Neither is a simple binary LZ4 compression, but a special compression integrated into the serialization pipeline, using MessagePackext code (Lz4BlockArray (98) orLz4Block (99)). Therefore, it is not readily compatible with compression offered in other languages.

Lz4Block compresses an entire MessagePack sequence as a single LZ4 block. This is the simple compression that achieves best compression ratio, at the cost of copying the entire sequence when necessary to get contiguous memory.

Lz4BlockArray compresses an entire MessagePack sequence as a array of LZ4 blocks. Compressed/decompressed blocks are chunked and thus do not enter the GC's Large-Object-Heap, but the compression ratio is slightly worse.

We recommend to useLz4BlockArray as the default when using compression.For compatibility with MessagePack v1.x, useLz4Block.

Regardless of which LZ4 option is set at the deserialization, both methods can be deserialized. For example, when theLz4BlockArray option was used, binary data using eitherLz4Block andLz4BlockArray can be deserialized. Neither can be decompressed and hence deserialized when the compression option is set toNone.

Attributions

LZ4 compression support is using Milosz Krajewski'slz4net code with some modifications.

Comparison with protobuf, JSON, ZeroFormatter

protobuf-net is major, widely used binary-format library on .NET. I love protobuf-net and respect their great work. But when you use protobuf-net as a general purpose serialization format, you may encounter an annoying issue.

[ProtoContract]publicclassParent{[ProtoMember(1)]publicintPrimitive{get;set;}[ProtoMember(2)]publicChildProp{get;set;}[ProtoMember(3)]publicint[]Array{get;set;}}[ProtoContract]publicclassChild{[ProtoMember(1)]publicintNumber{get;set;}}using(varms=newMemoryStream()){// serialize null.ProtoBuf.Serializer.Serialize<Parent>(ms,null);ms.Position=0;varresult=ProtoBuf.Serializer.Deserialize<Parent>(ms);Console.WriteLine(result!=null);// True, not null. but all property are zero formatted.Console.WriteLine(result.Primitive);// 0Console.WriteLine(result.Prop);// nullConsole.WriteLine(result.Array);// null}using(varms=newMemoryStream()){// serialize empty array.ProtoBuf.Serializer.Serialize<Parent>(ms,newParent{Array=System.Array.Empty<int>()});ms.Position=0;varresult=ProtoBuf.Serializer.Deserialize<Parent>(ms);Console.WriteLine(result.Array==null);// True, null!}

protobuf(-net) cannot handle null and empty collection correctly, because protobuf has nonull representation (seethis SO answer from a protobuf-net author).

MessagePack's type system can correctly serialize the entire C# type system. This is a strong reason to recommend MessagePack over protobuf.

Protocol Buffers have good IDL andgRPC support. If you want to use IDL, I recommendGoogle.Protobuf over MessagePack.

JSON is good general-purpose format. It is simple, human-readable and thoroughly-enough specified.Utf8Json - which I created as well - adopts same architecture as MessagePack for C# and avoids encoding/decoding costs as much as possible just like this library does. If you want to know more about binary vs text formats, seeUtf8Json/which serializer should be used.

ZeroFormatter is similar asFlatBuffers but specialized to C#, and special in that regard. Deserialization is infinitely fast but the produced binary size is larger. And ZeroFormatter's caching algorithm requires additional memory.

For many common uses, MessagePack for C# would be a better fit.

Hints to achieve maximum performance when using MessagePack for C#

MessagePack for C# prioritizes maximum performance by default. However, there are also some options that sacrifice performance for convenience.

Use indexed keys instead of string keys (Contractless)

TheDeserialization Performance for different options section shows the results of indexed keys (IntKey) vs string keys (StringKey) performance. Indexed keys serialize the object graph as a MessagePack array. String keys serializes the object graph as a MessagePack map.

For example this type is serialized to

[MessagePackObject]publicclassPerson{[Key(0)]or[Key("name")]public string Name{get;set;}[Key(1)]or[Key("age")]public int Age{get;set;}}newPerson{Name="foobar",Age=999}
  • IntKey:["foobar", 999]
  • StringKey:{"name:"foobar","age":999}.

IntKey is always fast in both serialization and deserialization because it does not have to handle and lookup key names, and always has the smaller binary size.

StringKey is often a useful, contractless, simple replacement for JSON, interoperability with other languages with MessagePack support, and less error prone versioning. But to achieve maximum performance, useIntKey.

Create own custom composite resolver

CompositeResolver.Create is an easy way to create composite resolvers. But formatter lookups have some overhead. If you create a custom resolver (or useStaticCompositeResolver.Instance), you can avoid this overhead.

publicclassMyApplicationResolver:IFormatterResolver{publicstaticreadonlyIFormatterResolverInstance=newMyApplicationResolver();// configure your custom resolvers.privatestaticreadonlyIFormatterResolver[]Resolvers=newIFormatterResolver[]{};privateMyApplicationResolver(){}publicIMessagePackFormatter<T>GetFormatter<T>(){returnCache<T>.Formatter;}privatestaticclassCache<T>{publicstaticIMessagePackFormatter<T>Formatter;staticCache(){// configure your custom formatters.if(typeof(T)==typeof(XXX)){Formatter=newICustomFormatter();return;}foreach(varresolverinResolvers){varf=resolver.GetFormatter<T>();if(f!=null){Formatter=f;return;}}}}}

NOTE: If you are creating a library, recommend using the above custom resolver instead ofCompositeResolver.Create. Also, libraries must not useStaticCompositeResolver - as it is global state - to avoid compatibility issues.

Use native resolvers

By default, MessagePack for C# serializes GUID as string. This is much slower than the native .NET format GUID. The same applies to Decimal. If your application makes heavy use of GUID or Decimal and you don't have to worry about interoperability with other languages, you can replace them with the native serializersNativeGuidResolver andNativeDecimalResolver respectively.

Also,DateTime is serialized using the MessagePack timestamp format. By using theNativeDateTimeResolver, it is possible to maintain Kind and perform faster serialization.

Be careful when copying buffers

MessagePackSerializer.Serialize returnsbyte[] in default. The finalbyte[] is copied from an internal buffer pool. That is an extra cost. You can useIBufferWriter<T> or theStream API to write to buffers directly. If you want to use a buffer pool outside of the serializer, you should implement customIBufferWriter<byte> or use an existing one such asSequence<T> from theNerdbank.Streams package.

During deserialization,MessagePackSerializer.Deserialize(ReadOnlyMemory<byte> buffer) is better than theDeserialize(Stream stream) overload. This is because the Stream API version starts by reading the data, generating aReadOnlySequence<byte>, and only then starts the deserialization.

Choosing compression

Compression is generally effective when there is duplicate data. In MessagePack, arrays containing objects using string keys (Contractless) can be compressed efficiently because compression can be applied to many duplicate property names. Indexed keys compression is not as effectively compressed as string keys, but indexed keys are smaller in the first place.

This is some example benchmark performance data;

SerializerMeanDataSize
IntKey2.941 us469.00 B
IntKey(Lz4)3.449 us451.00 B
StringKey4.340 us1023.00 B
StringKey(Lz4)5.469 us868.00 B

IntKey(Lz4) is not as effectively compressed, but performance is still somewhat degraded. On the other hand,StringKey can be expected to have a sufficient effect on the binary size. However, this is just an example. Compression can be quite effective depending on the data, too, or have little effect other than slowing down your program. There are also cases in which well-compressible data exists in the values (such as long strings, e.g. containing HTML data with many repeated HTML tags). It is important to verify the actual effects of compression on a case by case basis.

Extensions

MessagePack for C# has extension points that enable you to provide optimal serialization support for custom types. There are official extension support packages.

Install-Package MessagePack.ReactivePropertyInstall-Package MessagePack.UnityShimsInstall-Package MessagePack.AspNetCoreMvcFormatter

TheMessagePack.ReactiveProperty package adds support for types of theReactiveProperty library. It addsReactiveProperty<>,IReactiveProperty<>,IReadOnlyReactiveProperty<>,ReactiveCollection<>,Unit serialization support. It is useful for save viewmodel state.

TheMessagePack.UnityShims package provides shims forUnity's standard structs (Vector2,Vector3,Vector4,Quaternion,Color,Bounds,Rect,AnimationCurve,Keyframe,Matrix4x4,Gradient,Color32,RectOffset,LayerMask,Vector2Int,Vector3Int,RangeInt,RectInt,BoundsInt) and corresponding formatters. It can enable proper communication between servers and Unity clients.

After installation, extension packages must be enabled, by creating composite resolvers. Here is an example showing how to enable all extensions.

// Set extensions to default resolver.varresolver=MessagePack.Resolvers.CompositeResolver.Create(// enable extension packages firstReactivePropertyResolver.Instance,MessagePack.Unity.Extension.UnityBlitResolver.Instance,MessagePack.Unity.UnityResolver.Instance,// finally use standard (default) resolverStandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);// Pass options every time or set as defaultMessagePackSerializer.DefaultOptions=options;

For configuration details, see:Extension Point section.

TheMessagePack.AspNetCoreMvcFormatter is add-on forASP.NET Core MVC's serialization to boost up performance. This is configuration sample.

publicvoidConfigureServices(IServiceCollectionservices){services.AddMvc().AddMvcOptions(option=>{option.OutputFormatters.Clear();option.OutputFormatters.Add(newMessagePackOutputFormatter(ContractlessStandardResolver.Options));option.InputFormatters.Clear();option.InputFormatters.Add(newMessagePackInputFormatter(ContractlessStandardResolver.Options));});}

Other authors are creating extension packages, too.

  • MagicOnion - gRPC based HTTP/2 RPC Streaming Framework
  • MasterMemory - Embedded Readonly In-Memory Document Database

You can make your own extension serializers or integrate with frameworks. Let's create and share!

Experimental Features

MessagePack for C# has experimental features which provides you with very performant formatters. There is an official package.

Install-Package MessagePack.Experimental

For detailed information, see:Experimental.md

API

High-Level API (MessagePackSerializer)

TheMessagePackSerializer class is the entry point of MessagePack for C#. Static methods make up the main API of MessagePack for C#.

APIDescription
Serialize<T>Serializes an object graph to a MessagePack binary blob. Async variant for Stream available. Non-generic overloads available.
Deserialize<T>Deserializes a MessagePack binary to an object graph. Async variant for Stream available. Non-generic overloads available.
SerializeToJsonSerialize a MessagePack-compatible object graph to JSON instead of MessagePack. Useful for debugging.
ConvertToJsonConvert MessagePack binary to JSON. Useful for debugging.
ConvertFromJsonConvert JSON to a MessagePack binary.

TheMessagePackSerializer.Typeless class offers most of the same APIs as above, but removes all type arguments from the API, forcing serialization to include the full type name of the root object. It uses theTypelessContractlessStandardResolver. Consider the result to be a .NET-specific MessagePack binary that isn't readily compatible with MessagePack deserializers in other runtimes.

MessagePack for C# fundamentally serializes usingIBufferWriter<byte> and deserializes usingReadOnlySequence<byte> orMemory<byte>. Method overloads are provided to conveniently use it with common buffer types and the .NETStream class, but some of these convenience overloads require copying buffers once and therefore have a certain overhead.

The high-level API uses a memory pool internally to avoid unnecessary memory allocation. If result size is under 64K, it allocates GC memory only for the return bytes.

Each serialize/deserialize method takes an optionalMessagePackSerializerOptions parameter which can be used to specify a customIFormatterResolver to use or to activate LZ4 compression support.

Multiple MessagePack structures on a singleStream

To deserialize aStream that contains multiple consecutive MessagePack data structures,you can use theMessagePackStreamReader class to efficiently identify theReadOnlySequence<byte>for each data structure and deserialize it. For example:

staticasyncTask<List<T>>DeserializeListFromStreamAsync<T>(Streamstream,CancellationTokencancellationToken){vardataStructures=newList<T>();using(varstreamReader=newMessagePackStreamReader(stream)){while(awaitstreamReader.ReadAsync(cancellationToken)isReadOnlySequence<byte>msgpack){dataStructures.Add(MessagePackSerializer.Deserialize<T>(msgpack,cancellationToken:cancellationToken));}}returndataStructures;}

Low-Level API (IMessagePackFormatter<T>)

TheIMessagePackFormatter<T> interface is responsible for serializing a unique type. For exampleInt32Formatter : IMessagePackFormatter<Int32> represents Int32 MessagePack serializer.

publicinterfaceIMessagePackFormatter<T>{voidSerialize(refMessagePackWriterwriter,Tvalue,MessagePackSerializerOptionsoptions);TDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions);}

Many built-in formatters exists underMessagePack.Formatters. Your custom types are usually automatically supported with the built-in type resolvers that generate newIMessagePackFormatter<T> types on-the-fly using dynamic code generation. See ourAOT code generation support for platforms that do not support this.

However, some types - especially those provided by third party libraries or the runtime itself - cannot be appropriately annotated, and contractless serialization would produce inefficient or even wrong results.To take more control over the serialization of such custom types, write your ownIMessagePackFormatter<T> implementation.Here is an example of such a custom formatter implementation. Note its use of the primitive API that is described in the next section.

/// <summary>Serializes a <see cref="FileInfo" /> by its full path as a string.</summary>publicclassFileInfoFormatter:IMessagePackFormatter<FileInfo>{publicvoidSerialize(refMessagePackWriterwriter,FileInfovalue,MessagePackSerializerOptionsoptions){if(value==null){writer.WriteNil();return;}writer.WriteString(value.FullName);}publicFileInfoDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){if(reader.TryReadNil()){returnnull;}options.Security.DepthStep(refreader);varpath=reader.ReadString();reader.Depth--;returnnewFileInfo(path);}}

TheDepthStep andDepth-- statements provide a level of security while deserializing untrusted datathat might otherwise be able to execute a denial of service attack by sending MessagePack data that woulddeserialize into a very deep object graph leading to aStackOverflowException that would crash the process.This pair of statements should surround the bulk of anyIMessagePackFormatter<T>.Deserialize method.

Important: A message pack formatter mustread or write exactly one data structure.In the above example we just read/write a string. If you have more than one element to write out,you must precede it with a map or array header. You must read the entire map/array when deserializing.For example:

publicclassMySpecialObjectFormatter:IMessagePackFormatter<MySpecialObject>{publicvoidSerialize(refMessagePackWriterwriter,MySpecialObjectvalue,MessagePackSerializerOptionsoptions){if(value==null){writer.WriteNil();return;}writer.WriteArrayHeader(2);writer.WriteString(value.FullName);writer.WriteString(value.Age);}publicMySpecialObjectDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){if(reader.TryReadNil()){returnnull;}options.Security.DepthStep(refreader);stringfullName=null;intage=0;// Loop over *all* array elements independently of how many we expect,// since if we're serializing an older/newer version of this object it might// vary in number of elements that were serialized, but the contract of the formatter// is that exactly one data structure must be read, regardless.// Alternatively, we could check that the size of the array/map is what we expect// and throw if it is not.intcount=reader.ReadArrayHeader();for(inti=0;i<count;i++){switch(i){case0:fullName=reader.ReadString();break;case1:age=reader.ReadInt32();break;default:reader.Skip();break;}}reader.Depth--;returnnewMySpecialObject(fullName,age);}}

Your custom formatters must be discoverable via someIFormatterResolver. Learn more in ourresolvers section.

You can see many other samples frombuiltin formatters.

Primitive API (MessagePackWriter,MessagePackReader)

TheMessagePackWriter andMessagePackReader structs make up the lowest-level API. They read and write the primitives types defined in the MessagePack specification.

MessagePackReader

AMessagePackReader can efficiently read fromReadOnlyMemory<byte> orReadOnlySequence<byte> without any allocations, except to allocate a newstring as required by theReadString() method. All other methods return either value structs orReadOnlySequence<byte> slices for extensions/arrays.Reading directly fromReadOnlySequence<byte> means the reader can directly consume some modern high performance APIs such asPipeReader.

MethodDescription
SkipAdvances the reader's position past the current value. If the value is complex (e.g. map, array) the entire structure is skipped.
Read*Read and return a value whose type is named by the method name from the current reader position. Throws if the expected type does not match the actual type. When reading numbers, the type need not match the binary-specified type exactly. The numeric value will be coerced into the desired type or throw if the integer type is too small for a large value.
TryReadNilAdvances beyond the current value if the current value isnil and returnstrue; otherwise leaves the reader's position unchanged and returnsfalse.
ReadBytesReturns a slice of the input sequence representing the contents of abyte[], and advances the reader.
ReadStringSequenceReturns a slice of the input sequence representing the contents of astring without decoding it, and advances the reader.
CloneCreates a newMessagePackReader with the specified input sequence and the same settings as the original reader.
CreatePeekReaderCreates a new reader with the same position as this one, allowing the caller to "read ahead" without impacting the original reader's position.
NextCodeReads the low-level MessagePackbyte that describes the type of the next value. Does not advance the reader. SeeMessagePack format of first byte. Its static class hasToMessagePackType andToFormatName utility methods.MessagePackRange means Min-Max fix range of MessagePack format.
NextMessagePackTypeDescribes theNextCode value as a higher level category. Does not advance the reader. SeeMessagePack spec of source types.
(others)Other methods and properties as described by the .xml doc comment file and Intellisense.

TheMessagePackReader is capable of automatically interpreting both the old and new MessagePack spec.

MessagePackWriter

AMessagePackWriter writes to a given instance ofIBufferWriter<byte>. Several common implementations of this exist, allowing zero allocations and minimal buffer copies while writing directly to several I/O APIs includingPipeWriter.

TheMessagePackWriter writes the new MessagePack spec by default, but can write MessagePack compatible with the old spec by setting theOldSpec property totrue.

MethodDescription
CloneCreates a newMessagePackWriter with the specified underlyingIBufferWriter<byte> and the same settings as the original writer.
FlushWrites any buffered bytes to the underlyingIBufferWriter<byte>.
WriteNilWrites the MessagePack equivalent of .NET'snull value.
WriteWrites any MessagePack primitive value in the most compact form possible. Has overloads for every primitive type defined by the MessagePack spec.
Write*IntType*Writes an integer value in exactly the MessagePack type specified, even if a more compact format exists.
WriteMapHeaderIntroduces a map by specifying the number of key=value pairs it contains.
WriteArrayHeaderIntroduces an array by specifying the number of elements it contains.
WriteExtensionFormatWrites the full content of an extension value including length, type code and content.
WriteExtensionFormatHeaderWrites just the header (length and type code) of an extension value.
WriteRawCopies the specified bytes directly to the underlyingIBufferWriter<byte> without any validation.
(others)Other methods and properties as described by the .xml doc comment file and Intellisense.

DateTime is serialized toMessagePack Timestamp format, it serialize/deserialize UTC and losesKind info and requires thatMessagePackWriter.OldSpec == false.If you use theNativeDateTimeResolver,DateTime values will be serialized using .NET's nativeInt64 representation, which preservesKind info but may not be interoperable with non-.NET platforms.

Main Extension Point (IFormatterResolver)

AnIFormatterResolver is storage of typed serializers. TheMessagePackSerializer API accepts aMessagePackSerializerOptions object which specifies theIFormatterResolver to use, allowing customization of the serialization of complex types.

Resolver NameDescription
BuiltinResolverBuiltin primitive and standard classes resolver. It includes primitive(int, bool, string...) and there nullable, array and list. and some extra builtin types(Guid,Uri,BigInteger, etc...).
StandardResolverComposited resolver. It resolves in the following orderbuiltin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamic object fallback. This is the default of MessagePackSerializer.
ContractlessStandardResolverCompositedStandardResolver(except dynamic object fallback) ->DynamicContractlessObjectResolver ->DynamicObjectTypeFallbackResolver. It enables contractless serialization.
StandardResolverAllowPrivateSame as StandardResolver but allow serialize/deserialize private members.
ContractlessStandardResolverAllowPrivateSame as ContractlessStandardResolver but allow serialize/deserialize private members.
PrimitiveObjectResolverMessagePack primitive object resolver. It is used fallback inobject type and supportsbool,char,sbyte,byte,short,int,long,ushort,uint,ulong,float,double,DateTime,string,byte[],ICollection,IDictionary.
DynamicObjectTypeFallbackResolverSerialize is used type in fromobject type, deserialize is used PrimitiveObjectResolver.
AttributeFormatterResolverGet formatter from[MessagePackFormatter] attribute.
CompositeResolverComposes several resolvers and/or formatters together in an ordered list, allowing reuse and overriding of behaviors of existing resolvers and formatters.
NativeDateTimeResolverSerialize by .NET native DateTime binary format. It keepsDateTime.Kind that loses by standard(MessagePack timestamp) format.
NativeGuidResolverSerialize by .NET native Guid binary representation. It is faster than standard(string) representation.
NativeDecimalResolverSerialize by .NET native decimal binary representation. It is faster than standard(string) representation.
DynamicEnumResolverResolver of enum and there nullable, serialize there underlying type. It uses dynamic code generation to avoid boxing and boostup performance serialize there name.
DynamicEnumAsStringResolverResolver of enum and there nullable. It uses reflection call for resolve nullable at first time.
DynamicGenericResolverResolver of generic type(Tuple<>,List<>,Dictionary<,>,Array, etc). It uses reflection call for resolve generic argument at first time.
DynamicUnionResolverResolver of interface marked by UnionAttribute. It uses dynamic code generation to create dynamic formatter.
DynamicObjectResolverResolver of class and struct made by MessagePackObjectAttribute. It uses dynamic code generation to create dynamic formatter.
DynamicContractlessObjectResolverResolver of all classes and structs. It does not needsMessagePackObjectAttribute and serialized key as string(same as marked[MessagePackObject(true)]).
DynamicObjectResolverAllowPrivateSame as DynamicObjectResolver but allow serialize/deserialize private members.
DynamicContractlessObjectResolverAllowPrivateSame as DynamicContractlessObjectResolver but allow serialize/deserialize private members.
TypelessObjectResolverUsed forobject, embed .NET type in binary byext(100) format so no need to pass type in deserialization.
TypelessContractlessStandardResolverComposited resolver. It resolves in the following ordernativedatetime -> builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamiccontractless -> typeless. This is the default ofMessagePackSerializer.Typeless

Each instance ofMessagePackSerializer accepts only a single resolver. Most object graphs will need more than one for serialization, so composing a single resolver made up of several is often required, and can be done with theCompositeResolver as shown below:

// Do this once and store it for reuse.varresolver=MessagePack.Resolvers.CompositeResolver.Create(// resolver custom types firstReactivePropertyResolver.Instance,MessagePack.Unity.Extension.UnityBlitResolver.Instance,MessagePack.Unity.UnityResolver.Instance,// finally use standard resolverStandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);// Each time you serialize/deserialize, specify the options:byte[]msgpackBytes=MessagePackSerializer.Serialize(myObject,options);TmyObject2=MessagePackSerializer.Deserialize<MyObject>(msgpackBytes,options);

A resolver can be set as default withMessagePackSerializer.DefaultOptions = options, butWARNING:When developing an application where you control all MessagePack-related code it may be safe to rely on this mutable static to control behavior.For all other libraries or multi-purpose applications that useMessagePackSerializer you should explicitly specify theMessagePackSerializerOptions to use with each method invocation to guarantee your code behaves as you expect even when sharing anAppDomain or process with other MessagePack users that may change this static property.

Here is sample of useDynamicEnumAsStringResolver withDynamicContractlessObjectResolver (It is Json.NET-like lightweight setting.)

// composite same as StandardResolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(MessagePack.Resolvers.BuiltinResolver.Instance,MessagePack.Resolvers.AttributeFormatterResolver.Instance,// replace enum resolverMessagePack.Resolvers.DynamicEnumAsStringResolver.Instance,MessagePack.Resolvers.DynamicGenericResolver.Instance,MessagePack.Resolvers.DynamicUnionResolver.Instance,MessagePack.Resolvers.DynamicObjectResolver.Instance,MessagePack.Resolvers.PrimitiveObjectResolver.Instance,// final fallback(last priority)MessagePack.Resolvers.DynamicContractlessObjectResolver.Instance);

If you want to make an extension package, you should write both a formatter and resolverfor easier consumption.Here is sample of a resolver:

publicclassSampleCustomResolver:IFormatterResolver{// Resolver should be singleton.publicstaticreadonlyIFormatterResolverInstance=newSampleCustomResolver();privateSampleCustomResolver(){}// GetFormatter<T>'s get cost should be minimized so use type cache.publicIMessagePackFormatter<T>GetFormatter<T>(){returnFormatterCache<T>.Formatter;}privatestaticclassFormatterCache<T>{publicstaticreadonlyIMessagePackFormatter<T>Formatter;// generic's static constructor should be minimized for reduce type generation size!// use outer helper method.staticFormatterCache(){Formatter=(IMessagePackFormatter<T>)SampleCustomResolverGetFormatterHelper.GetFormatter(typeof(T));}}}internalstaticclassSampleCustomResolverGetFormatterHelper{// If type is concrete type, use type-formatter mapstaticreadonlyDictionary<Type,object>formatterMap=newDictionary<Type,object>(){{typeof(FileInfo),newFileInfoFormatter()}// add more your own custom serializers.};internalstaticobjectGetFormatter(Typet){objectformatter;if(formatterMap.TryGetValue(t,outformatter)){returnformatter;}// If type can not get, must return null for fallback mechanism.returnnull;}}

MessagePackFormatterAttribute

MessagePackFormatterAttribute is a lightweight extension point of class, struct, interface, enum and property/field. This is like Json.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter.

[MessagePackFormatter(typeof(CustomObjectFormatter))]publicclassCustomObject{stringinternalId;publicCustomObject(){this.internalId=Guid.NewGuid().ToString();}// serialize/deserialize internal field.classCustomObjectFormatter:IMessagePackFormatter<CustomObject>{publicvoidSerialize(refMessagePackWriterwriter,CustomObjectvalue,MessagePackSerializerOptionsoptions){options.Resolver.GetFormatterWithVerify<string>().Serialize(refwriter,value.internalId,options);}publicCustomObjectDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){varid=options.Resolver.GetFormatterWithVerify<string>().Deserialize(refreader,options);returnnewCustomObject{internalId=id};}}}// per field, memberpublicclassInt_x10Formatter:IMessagePackFormatter<int>{publicintDeserialize(refMessagePackReaderreader,MessagePackSerializerOptionsoptions){returnreader.ReadInt32()*10;}publicvoidSerialize(refMessagePackWriterwriter,intvalue,MessagePackSerializerOptionsoptions){writer.WriteInt32(value*10);}}[MessagePackObject]publicclassMyClass{// You can attach custom formatter per member.[Key(0)][MessagePackFormatter(typeof(Int_x10Formatter))]publicintMyProperty1{get;set;}}

Formatter is retrieved byAttributeFormatterResolver, it is included inStandardResolver.

IgnoreFormatter

IgnoreFormatter<T> is lightweight extension point of class and struct. If there exists types that can't be serialized, you can registerIgnoreFormatter<T> that serializes those to nil/null.

// CompositeResolver can set custom formatter.varresolver=MessagePack.Resolvers.CompositeResolver.Create(newIMessagePackFormatter[]{// for example, register reflection infos (can not serialize)newIgnoreFormatter<MethodBase>(),newIgnoreFormatter<MethodInfo>(),newIgnoreFormatter<PropertyInfo>(),newIgnoreFormatter<FieldInfo>()},newIFormatterResolver[]{ContractlessStandardResolver.Instance});

Reserved Extension Types

MessagePack for C# already used some MessagePack extension type codes, be careful to avoid using the same ext code for other purposes.

RangeReserved for
[-128, -1]Reserved by the msgpack spec for predefined types
[30, 120)Reserved for this library's use to support common types in .NET

This leaves the following ranges for your use:

  • [0, 30)
  • [120, 127]

Within thereserved ranges, this library defines or implements extensions that use these type codes:

CodeTypeUse by
-1DateTimeMessagePack-spec reserved for timestamp
30Vector2[]for Unity, UnsafeBlitFormatter
31Vector3[]for Unity, UnsafeBlitFormatter
32Vector4[]for Unity, UnsafeBlitFormatter
33Quaternion[]for Unity, UnsafeBlitFormatter
34Color[]for Unity, UnsafeBlitFormatter
35Bounds[]for Unity, UnsafeBlitFormatter
36Rect[]for Unity, UnsafeBlitFormatter
37Int[]for Unity, UnsafeBlitFormatter
38Float[]for Unity, UnsafeBlitFormatter
39Double[]for Unity, UnsafeBlitFormatter
98AllMessagePackCompression.Lz4BlockArray
99AllMessagePackCompression.Lz4Block
100objectTypelessFormatter

Unity support

The minimum supported Unity version will be2022.3.12f1, as it is necessary to support IL2CPP via C# Source Generator.

There are two installation steps required to use it in Unity. Do both, not just one.

  1. InstallMessagePack from NuGet usingNuGetForUnityOpen Window from NuGet -> Manage NuGet Packages, Search "MessagePack" and Press Install.

  2. InstallMessagePack.Unity package by referencing the git URL.Open Package Manager window and pressAdd Package from git URL..., enter following path

    https://github.com/MessagePack-CSharp/MessagePack-CSharp.git?path=src/MessagePack.UnityClient/Assets/Scripts/MessagePack

    MessagePack uses the ..* release tag, so you can specify a version like #v3.0.0. For example:https://github.com/MessagePack-CSharp/MessagePack-CSharp.git?path=src/MessagePack.UnityClient/Assets/Scripts/MessagePack#v3.0.0

In Unity, MessagePackSerializer can serializeVector2,Vector3,Vector4,Quaternion,Color,Bounds,Rect,AnimationCurve,Keyframe,Matrix4x4,Gradient,Color32,RectOffset,LayerMask,Vector2Int,Vector3Int,RangeInt,RectInt,BoundsInt and their nullable, array and list types with the built-in extensionUnityResolver.

MessagePack.Unity automatically addsUnityResolver to the default options Resolver when the application starts with code like this in the unity package to enable this serialization:

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.SubsystemRegistration)]privatestaticvoidInit(){MessagePackSerializer.DefaultOptions=MessagePackSerializerOptions.Standard.WithResolver(UnityResolver.InstanceWithStandardResolver);}

If you want to customize the Resolver or change the DefaultOptions, it would be good to keep this in mind.

Share types with .NET

TheMessagePack.UnityShims NuGet package is for .NET server-side serialization support to communicate with Unity. It includes shims for Vector3 etc and the Safe/Unsafe serialization extension.

There are several ways to share types between .NET and Unity:

  • Share using symbolic links
  • Place the actual files on the Unity side and reference them as link files in the .NET csproj
  • Use UPM local references to reference the .NET project from the Unity side

While the setup is a bit challenging, the smoothest way to share is using UPM local references. For detailed steps, please refer to theMagicOnion Sample.

UnsafeBlitResolver

MessagePack for C# has an additional unsafe extension.UnsafeBlitResolver is special resolver for extremely fast but unsafe serialization/deserialization of struct arrays.

image

x20 faster Vector3[] serialization than native JsonUtility. If useUnsafeBlitResolver, serialization uses a special format (ext:typecode 30~39) forVector2[],Vector3[],Quaternion[],Color[],Bounds[],Rect[]. If useUnityBlitWithPrimitiveArrayResolver, it supportsint[],float[],double[] too. This special feature is useful for serializing Mesh (manyVector3[]) or many transform positions.

If you want to use unsafe resolver, registerUnityBlitResolver orUnityBlitWithPrimitiveArrayResolver.

Here is sample of configuration.

StaticCompositeResolver.Instance.Register(MessagePack.Unity.UnityResolver.Instance,MessagePack.Unity.Extension.UnityBlitWithPrimitiveArrayResolver.Instance,MessagePack.Resolvers.StandardResolver.Instance);varoptions=MessagePackSerializerOptions.Standard.WithResolver(StaticCompositeResolver.Instance);MessagePackSerializer.DefaultOptions=options;

AOT Code Generation

A source generator is provided in theMessagePackAnalyzer package, which is automatically installed when you installMessagePack via NuGet.This will source generate the formatters required for all your[MessagePackObject]-annotated data types during compilation for the fastest possible startup and runtime.AnIFormatterResolver is also generated that bundles all source generated and user-written formatters together.TheStandardResolver includes theSourceGeneratedFormatterResolver which discovers and uses your source generated resolver automatically.

Therefore, in the usual scenario, it will work with AOT Safe without any special handling.If you prefer to restrict your resolver to source-generated formatters, you should use theMessagePack.GeneratedMessagePackResolver, which is source generated into your project for that purpose.This type's name and namespace can be customized by applying[GeneratedMessagePackResolver] to apartial class that you define, at which point that class becomes the resolver for you to use.

At runtime, if a source generated or hand-written formatter cannot be found for a given[MessagePackObject] type, MessagePack will generate the formatters on the fly usingReflection.Emit to create highly-tuned formatters for each type.This code generation has a minor upfront performance cost.

Note: When using Unity, dynamic code generation only works when targeting .NET Framework 4.x + mono runtime.For all other Unity targets, AOT is required.

Customizations

You can customize the generated source through properties on theGeneratedMessagePackResolverAttribute.

[GeneratedMessagePackResolver]partialclassMyResolver{}

When exposing the generated resolver publicly, consumers outside the library should aggregate the resolver using itsInstance property, which containsonly the generated formatters.

Two assembly-level attributes exist to help with mixing in your own custom formatters with the automatically generated ones:

  • MessagePackKnownFormatterAttribute
  • MessagePackAssumedFormattableAttribute

Learn more about using a mix of your own custom formatters and automatically generated ones inthe Analyzer section.

MagicOnion

MagicOnion is a code-first gRPC framework based on grpc-dotnet and MessagePack. gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not need IDL. When communicating C# to C#, schemaless (or rather C# classes as schema) is better than using IDL.

StreamJsonRpc

The StreamJsonRpc library is based onJSON-RPC and includesa pluggable formatter architecture and as of v2.3 includesMessagePack support.

How to build

See ourcontributor's guide.

Code of Conduct

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community.For more information see the.NET Foundation Code of Conduct.

.NET Foundation

This project is supported by the.NET Foundation.

About

Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#]

Topics

Resources

License

Contributing

Security policy

Stars

Watchers

Forks

Sponsor this project

  •  
  •  

Packages

No packages published

Languages


[8]ページ先頭

©2009-2025 Movatter.jp