> Use Thrift if you're developing RPC services.
There's gRPC now, which uses Protobuf messages and is much better than Thrift.
Data serialization
41–50 of 91 posts
Re: Data serialization
#42Protobufs and similar projects like it (gRPC, Cap'n Proto) seem really interesting, but I haven't come across a time at work yet where it's made sense to the team to adopt it. Maybe that's just my own inexperience, but the serialization scheme is low on the list compared to optimizing DB queries, getting rid of bloat in the app, etc. I've been waiting for an excuse to adopt this stuff at work because it seems really…
Defining your schema upfront and have types generated in multiple languages is the real value add. The performance is a nice cherry on top.
More specifically, being able to agree and communicate schemas to other collaborators is a big win when trying to avoid interfacing bugs.
OTOH schemas seem less necessary if you are working on a completely self-contained app.
Re: Data serialization
#43Earlier quoted context omitted.
Just to add another dimension to your analysis, if you're sending large binaries the 33% overhead of Base64 adds up pretty quickly. It may not apply to your use case but if you are delivering images or video over a websocket it can make a big difference.
On the other hand, gzip can reclaim most of those 33% with Huffman encoding.
Only in some cases. Also it increases latency.
Here’s my old article about Microsoft’s binary XML format: http://const.me/articles/net-tcp/ As you see, for some payloads it compresses XML to 9% of the original size.
The data contract serializer included in .NET framework (and even in .NET Core) can read & write objects directly from/to this binary serialization format, without intermediate text XML anywhere.
Re: Data serialization
#44Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…
I keep reiterating that there was nothing wrong with XML.
Or ASN.1.
Re: Data serialization
#45Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…
> I hate those "schemaless" (aka schema-on-read) serialization formats more and more
This, and also I think a lot of people who preach the flexibility of schemaless anything really just wants a more flexible schema definition language, with a lot of "maybe" options and similar stuff.
Re: Data serialization
#46I would also recommend CBOR (http://cbor.io/); one can think of it like a binary form of JSON. It has a few advantages:
* datetime objects
* binary blobs
Re: Data serialization
#47OT, but I'm excited to see TiddlyWiki in the wild.
Re: Data serialization
#48> Use Thrift if you're developing RPC services. There's gRPC now, which uses Protobuf messages and is much better than Thrift. https://grpc.io
Could you expand on that a bit? I'm interested in promoting gRPC over thrift at work, but so far the only benefits I see are the HTTP/2 transport which allows for better load balancing and request tracing on the transport level.
Re: Data serialization
#49Serialization format has no relation with schemas. Also, Schema validation can be as strict or lax as you want it to be. I wrote Spyne (in Python) mainly to abstract the schema from the serialization format. The protocols are totally pluggable and your code cares only about the models and not the serialization format. The latest alpha is out not long ago. Check it out if this sounds interesting. https://github.com/ar…
This sounds interesting and I would have liked to read more but currently I'm getting timeout issues trying to visit spyne.io. Could you please check?