A successor to MessagePack-CSharp

[AArnott]

The Nerdbank.MessagePack library is (in my mind) the successor to MessagePack-CSharp.

You can read more about this transition in my blog post.

See how this library has so many features that MessagePack-CSharp has oft been requested to add but never did.

[AArnott]

@aradalvand this new project is why I'm not spending a lot more time on MessagePack-CSharp. I hope you like the successor.

[AArnott]

I documented some Vogen workarounds: https://aarnott.github.io/Nerdbank.MessagePack/docs/type-shapes.html?tabs=net#working-with-vogen

My favorite is your custom marshaler idea, which I've documented and filed SteveDunn/Vogen#834 to try to get it built into Vogen itself.

[AArnott]

If you're calling Type.MakeGenericType at runtime, then you're already limited to dynamic runtimes (i.e. strike out NativeAOT as a possibility), so in that case there's little or no loss in migrating from MessagePack-CSharp to Nerdbank.MessagePack using ReflectionTypeShapeProvider.Default.

I expect throughput performance will still be decent, and while startup performance won't be as good as with the source generated shape provider, it should be quite a bit better than MessagePack-CSharp (without v3 source generation) due to much less use of Reflection.Emit. And compared to MessagePack-CSharp with v3 source generation... it may be somewhat slower on startup, I'm not sure.

these types are effectively determined at runtime because the code that supplies the concrete type arguments for these types is itself generic and registered as an open generic service in the DI container

If you knew what these generic type arguments tended to be though, you could still use the GenerateShapeFor<T> attribute to use the source generator. If the DI container ever changed which type arguments you needed, the type shapes wouldn't be available until a new or adjusted GenerateShapeFor<T> attribute was added to the witness class.

[aradalvand]

If you're calling Type.MakeGenericType at runtime, then you're already limited to dynamic runtimes (i.e. strike out NativeAOT as a possibility), so in that case there's little or no loss in migrating from MessagePack-CSharp to Nerdbank.MessagePack using ReflectionTypeShapeProvider.Default.

Fair point. However, even in some cases where the entire set of possible instantiations is statically known, I don't think there's a way to automatically inform PolyType of that, bar just writing them by hand. Is there?
I thought about building a source generator that would emit something like this:

[GenerateShapeFor<DeferredEventWorkItem<KnownRoot1, KnownId1, KnownEvent1>>]
[GenerateShapeFor<DeferredEventWorkItem<KnownRoot2, KnownId2, KnownEvent2>>]
// ...
public partial class Witness;

Of course, the problem is, once again, the dreaded dotnet/roslyn#57239
It's difficult to overstate how big of an impediment that single issue is to so many powerfully expressive source generation scenarios.

[AArnott]

There's only one way I can think of to 'automatically' tell PolyType about it. It uses polytype "associated types".

Essentially, if you have some X<T> that you have a shape for, you can ask PolyType to be sure to also emit some Y<T> shape as well. The generic arity in that example is 1 and in your example is 3, but it still works.

But this trick is only useful if you already have (or can create) some generic type that would get a shape already for whatever reason. That's admittedly a slim chance.

The only other workaround is to write it out by hand. And it doesn't have to be as an attribute directly on the Witness type either. It could be as a phantom field on some class that you already get a shape for.

[aradalvand]

@AArnott I want to commend you for the efforts you've put into Nerdbank.MessagePack; excellent job. The migration from MessagePack went more smoothly and with fewer gnarly issues along the way than I was anticipating. I particularly loved the runtime derived type registration feature, which I had a need for but didn't expect to find such a clean solution for in the new library. Also, the MessagePackConverter analyzers were quite impressive.

I'll let you know if I run into any severe limitations in the library going forward; but it's been great so far. Thank you.

[AArnott]

@Seadoo2017 MessagePack-CSharp hasn't been getting much love lately, as I've been quite busy, and the co-owner doesn't spend much time on it either. I see you reported a bug that you're hoping will get fixed. I'm not likely to get to it unless it bites me personally or I'm hired to fix it.

I hope you'll have much better luck with MessagePack-CSharp's successor, Nerdbank.MessagePack, which I'm very actively working on and supporting as bugs come in. See the original post above for details. Let me know how it goes.

[AArnott]

@koopmansrw I saw recently that you're trying multiple messagepack libraries. I'd love to invite you to try this one. :)

[koopmansrw]

First reaction is that it´s quite easy to use, and I like the DeserializeDynamicPrimitives. Unfortunately, I still get faulty data in one of my fields This is an array of string representing hexadecimal/binary data. The issue is that, in some cases, I get replacement characters, corrupting my data: "�\0�\u0002X\u0002" whereas I'd expect this value to be "0xC000BE025802". MsgPack.Cli actually gets the correct value for this field, but messes up the other fields (that both MessagePack and this one do get correct).

Basically it consists of an array of data.. Where each array entry consists of 6 - 42 bytes of data. In some cases it's presented as a something like "\u001d<*\0\u0013\023##################", but in other cases, I get it with replacement characters.. I'm using Flags to determine which kind of data I'm getting.

[AArnott]

If you'll send me a sample msgpack structure (hex or base64 encoded is fine, or just attach a raw file) I'll be happy to take a look. If we're corrupting the data I definitely want to fix that. But my guess is that either the original data isn't msgpack compliant, or your C# type definition doesn't match what it needs to be. Either way, I hope to help you out.

[koopmansrw]

I appreciate it a lot, I've sent you an e-mail as I'm a bit hesitant to share too much details on GitHub.

[AArnott]

(for other viewers) The msgpack data was written according to ye old spec which had no encoding for binary data. As a result, your data (like others) reused the String header for binary data, which means that parsers with no additional type information try to decode the binary data as UTF-8, but it isn't valid UTF-8 so it gets corrupted.

To make this work you have to add the type information by declaring a type to deserialize into, and use byte[] where the "string" data is. Then it will read it as binary.

At least, that was the idea. But we had an issue where our byte[] converter wouldn't read Str header data. I fix that here: #610

[AArnott]

Now available in 0.9.35-rc

[AArnott]

@kukuro-kt I know you were exploring deserializing a msgpack array of bytes recently. This works today in Nerdbank.MessagePack:

[Fact]
public void Bug2230()
{
	MessagePackSerializer serializer = new();
	byte[] msgpack = [145, 147, 127, 127, 127];
	Ms value = serializer.Deserialize<Ms>(msgpack);
	Assert.Equal(new byte[] { 127, 127, 127 }, value.rgb);
}

[GenerateShape]
public partial class Ms
{
	[Key(0)] public byte[] rgb;
}

See the OP of this page for details of this new library.

[kukuro-kt]

I'm starting use it on Unity, but it says "Cannot resolve method":


Did I get something wrong?

[AArnott]

Hmm... well first off I'll offer this unity-specific doc if you haven't discovered it yet.

As for this particular issue, it looks apparent to me... the "candidates are" list doesn't show an overload that matches what you're doing. All of them take 3 parameters. Now, the 3rd parameter being CancellationToken is optional. But the second one is not.

Now, what you're doing should actually work with the very latest version of this library (released yesterday). I expect you're using that version already. But your candidates list is missing some options that I think would make your code OK to compile. I can't tell what your using list is, but based on your fully-qualified Nerdbank.MessagePack.MessagePackSerializer bit of code, I'm guessing you're missing a using Nerdbank.MessagePack; at the top of your file.

Per these remarks:

When targeting .NET Standard 2.0 or .NET Framework, some important methods are available only as extension methods, so make sure to have a using Nerdbank.MessagePack; directive in your code file to see these.

[kukuro-kt]

Thanks! I misunderstood some parts in doc. I have fixed it with your guidance. byte[] field really works well here as you said🥳.

[kukuro-kt]

@AArnott Do you consider adding this project to MsgPack Org Homepage?
I may never have chance to know this project until you told me. I knew MessagePack-CShape from there, I think many people were same as me. If that website includes this project, more people will know this project.

[AArnott]

I have tried but I can't find anybody who maintains that page anymore and following the instructions that were written years ago to get yourself added automatically doesn't work anymore.

[kukuro-kt]

I have tried but I can't find anybody who maintains that page anymore and following the instructions that were written years ago to get yourself added automatically doesn't work anymore.

Have you asked https://x.com/frsyuki ? He is the creator & maintainner. I found he is still playing twitter actively recently.

[AArnott]

Thank you so much for the tip. I'll reach out.

[AArnott]

@treziac It sounds like you might be interested in this successor to MessagePack-CSharp, since it doesn't require attributes at all by default. Nerdbank.MessagePack has STJ-like contractless behavior, where attributes can be added to override the defaults.

[treziac]

Thanks!
Would love to give a try, but we have some strict policies on usage of third party nuget at work, especially on serialization. Unless there is a direct notification from messagepack-csharp to do the move and a sufficient reference on various sites, I won't be able to make the change.

Will definitely test it on personal projects on similar use case issues we face at work to give some feedbacks though

[AArnott]

That makes sense. I actually wrote much of MessagePack-CSharp (as you'll see from commit history) but my co-owner doesn't want me to use that repo's platform to point people to my new library. So I think the only 'direct message' you'll get from that repo is looking at how slow it moves and how many active issues sit there without getting any attention, since the co-owner doesn't spend much time on it.

I understand how work policies can influence these decisions. FWIW though, I think it will be within the spirit of work policies if you try to make the case on the basis that:

1. The original project is poorly supported nowadays.
2. The new project is very actively maintained by one of the two main contributors to the original.
3. That maintainer (me) is a Microsoft employee, and while Microsoft does not officially endorse this library, the Microsoft Visual Studio product is about to take a significant dependency on it.

The usability improvements, the unique features, etc. etc. of the new library won't likely persuade them, but the simple points above should. Hopefully they will.

[AArnott]

@PawelGerr I saw you are working with MessagePack-CSharp and recently had an issue with a stack overflow in one of its analyzers. I can tell you as one of the two maintainers of that repo, neither of us are paying much attention to it these days. I invite you to check out this library (see original post) which I'm actively working on. It has many new features and more and better analyzers, as you will see.
I'd love to hear your feedback.

[AArnott]

@Nikodage I didn't want to mention it on the old MessagePack-CSharp repo, but as one of its two chief contributors, I've moved on to this repo, which is the successor to MessagePack-CSharp. I don't spend much time on the former library any more. And FWIW the other contributor doesn't either. This library is very much maintained and has a lot more features than the former library.
Learn more about it on the top post on this page.

I'd love to get your feedback on it.

[AArnott]

@ThomasBleijendaal I see you've been having trouble with MessagePack-CSharp. Would you care to evaluate its successor? See the top post on this page for more.

[AArnott]

@jcasts This library will let you unload types. It doesn't store anything in mutable statics, so you control the cache lifetime by the MessagePackSerializer instance as you suggested with the older library.

[eiriktsarpalis]

@eiriktsarpalis Are these static properties within PolyType?

Yes, ReflectionTypeShapeProvider does define singleton instances and static caches of caches (primarily through ReflectionTypeShapeProvider.Default). I think the issue here though primarily is with the instances themselves which have not been designed with type unloading in mind, since individual ITypeShape instances are cached on a concurrent dictionary rather than a conditional weak table. If you can file an issue on the PolyType repo we can try to address that.

[AArnott]

eiriktsarpalis/PolyType#385

[jcasts]

I've tested assigning a new ReflectionTypeShapeProvider to each dynamically loaded assembly and only keeping it in memory with the assembly. This does seem to stabilize memory usage. I also had to dynamically create some lambdas to call the Serialize and Deserialize methods since I can't cast ITypeShape to a known ITypeShape<T>.
https://gist.github.com/jcasts/fa6a3733c5038a7209e419ce6aa66d60

[jcasts]

I am using .Net and using AssemblyLoadContext so that would be an ideal domain for caching.

[AArnott]

Per the PolyType resolution and your comment there, the issue seems to be resolved and I'm considering the matter closed. Thank you for using Nerdbank.MessagePack and for your feedback to help make it (and PolyType, in this case) better.

[AArnott]

@ShadowMarker789 The scoped modifier you're looking for already exists in the newer library. See the original post above for details.

[ShadowMarker789]

I happen to be using MessagePack through ASP Net Core SignalR Hubs.

Is there a path of low resistance to using this variant in that usage environment?

[AArnott]

Yes, this new library has SignalR support.