11–20 of 91 posts
Regarding JSON, this page says: > Performance is not good when dataset is huge. Program usually needs to load all data into memory first. This is just downright false. There are plenty of SAX-style JSON parsers.
In both cases I'm using MsgPack. It was easy enough to implement and I like how it has real numeric types.
Protobufs and similar projects like it (gRPC, Cap'n Proto) seem really interesting, but I haven't come across a time at work yet where it's made sense to the team to adopt it. Maybe that's just my own inexperience, but the serialization scheme is low on the list compared to optimizing DB queries, getting rid of bloat in the app, etc. I've been waiting for an excuse to adopt this stuff at work because it seems really…
Defining your schema upfront and have types generated in multiple languages is the real value add. The performance is a nice cherry on top.
It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it. Tab-separated-values are just data fields separated by a distinguished character, with rows separated by another distinguished character; as simple as it gets and no API required.
I read somewhere the problem with MsgPack is, that JSON has a rather fast parser build into JavaScript that beats the MsgPack parser. So you would have to check if the saved bandwidth would be enough to justify the slower parsing. Would be interesting if this still holds true with a WASM implementation.
I used to think binary formats for network protocols were a really good idea until I found out how big the TCP header itself can be. Saving a few bytes from your payload by using binary representations of integers doesn't make a huge difference when the TCP header is 60 bytes.
I wrote Spyne (in Python) mainly to abstract the schema from the serialization format. The protocols are totally pluggable and your code cares only about the models and not the serialization format. The latest alpha is out not long ago. Check it out if this sounds interesting.
https://github.com/arskom/spyne
Code generator: http://spyne.io
http://wiki.c2.com/?XmlIsaPoorCopyOfEssExpressions
They have a canonical representation
https://en.wikipedia.org/wiki/Canonical_S-expressions
I swear I have seen a proposal for an efficient binary representation somewhere but I can't find it.
Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…
- Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys)
- A standard Date format
- Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding.
- Arbitrary precision integer support. This is particularly useful for cryptocurrencies and for interoperability with 64 bit integers in other languages. And bigints are coming to javascript - https://github.com/tc39/proposal-bigint
- Maybe even fix JSON's weird unicode encoding: http://timelessrepo.com/json-isnt-a-javascript-subset
Unfortunately it seems like nobody 'owns' JSON enough to give JSON 2.0 the political weight it would need for cross-language support.
Earlier quoted context omitted.
I used to think binary formats for network protocols were a really good idea until I found out how big the TCP header itself can be. Saving a few bytes from your payload by using binary representations of integers doesn't make a huge difference when the TCP header is 60 bytes.
Just to add another dimension to your analysis, if you're sending large binaries the 33% overhead of Base64 adds up pretty quickly. It may not apply to your use case but if you are delivering images or video over a websocket it can make a big difference.