Data serialization
enqueuezero.com
Data serialization
1–10 of 91 posts
Re: Data serialization
#2Protobufs and similar projects like it (gRPC, Cap'n Proto) seem really interesting, but I haven't come across a time at work yet where it's made sense to the team to adopt it. Maybe that's just my own inexperience, but the serialization scheme is low on the list compared to optimizing DB queries, getting rid of bloat in the app, etc. I've been waiting for an excuse to adopt this stuff at work because it seems really cool!
Re: Data serialization
#3Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That is an important dimension to compare serialization on, and JSON is a complete loser in that sense. Not to mention how it flawed its numeric type is.
Re: Data serialization
#4Protobufs and similar projects like it (gRPC, Cap'n Proto) seem really interesting, but I haven't come across a time at work yet where it's made sense to the team to adopt it. Maybe that's just my own inexperience, but the serialization scheme is low on the list compared to optimizing DB queries, getting rid of bloat in the app, etc. I've been waiting for an excuse to adopt this stuff at work because it seems really…
Defining your schema upfront and have types generated in multiple languages is the real value add. The performance is a nice cherry on top.
Re: Data serialization
#5Regarding JSON, this page says:
> Performance is not good when dataset is huge. Program usually needs to load all data into memory first.
This is just downright false. There are plenty of SAX-style JSON parsers.
Re: Data serialization
#6I read somewhere the problem with MsgPack is, that JSON has a rather fast parser build into JavaScript that beats the MsgPack parser. So you would have to check if the saved bandwidth would be enough to justify the slower parsing.
Would be interesting if this still holds true with a WASM implementation.
Re: Data serialization
#7I read somewhere the problem with MsgPack is, that JSON has a rather fast parser build into JavaScript that beats the MsgPack parser. So you would have to check if the saved bandwidth would be enough to justify the slower parsing. Would be interesting if this still holds true with a WASM implementation.
I used to think binary formats for network protocols were a really good idea until I found out how big the TCP header itself can be. Saving a few bytes from your payload by using binary representations of integers doesn't make a huge difference when the TCP header is 60 bytes.
Re: Data serialization
#8re: protobufs
> It requires the program doing data parsing work to have the generated library as well. This would generally cause problem when schema modified.
That's not the case, and is the exact reason why you need to specify tag numbers in protos - so that you can make your schema forward and backward compatible when decoding/encoding from/to the wire format.
Re: Data serialization
#9Here are some benchmarks for Python and Java https://stackoverflow.com/questions/9884080/fastest-packing-...
Re: Data serialization
#10I read somewhere the problem with MsgPack is, that JSON has a rather fast parser build into JavaScript that beats the MsgPack parser. So you would have to check if the saved bandwidth would be enough to justify the slower parsing. Would be interesting if this still holds true with a WASM implementation.
Since WASM can't create or access JavaScript objects directly, I bet it's still slower. MessagePack (or my preference, CBOR) is still useful and faster than JSON if you need to exchange raw bytes from typed arrays (for JSON you'd need to encode them, e.g. in base64).