Live data from Hacker News

Data serialization

enqueuezero.com

71–80 of 91 posts

Re: Data serialization

#71
post #59
post #50

Earlier quoted context omitted.

Or S-expressions.

Both ASN.1 and XML are standardized (with extensive, lengthy and detailed specifications!). The term "S-expression" covers a very broad family of vaguely related not-very-interoperable surface syntaxes.

S-expressions are standardized in both Common Lisp and Scheme. The two standards are not identical, but are mostly compatible, with the CL standard being mostly a superset of the Scheme standard.

Re: Data serialization

#72
post #52
post #19

Earlier quoted context omitted.

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?

you want to use protobufs

Re: Data serialization

#73
post #19
post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

protobufs already support all these

Re: Data serialization

#74

> Use Thrift if you're developing RPC services. There's gRPC now, which uses Protobuf messages and is much better than Thrift. https://grpc.io

Could you expand on that a bit? I'm interested in promoting gRPC over thrift at work, but so far the only benefits I see are the HTTP/2 transport which allows for better load balancing and request tracing on the transport level.

gRPC with Protobuf has been faster in our usage and also has more development these days compared to Thrift. It's also simpler and better designed by just sticking to a single well-tested serialization format.

HTTP/2 is also a big advantage since it's standardized and easily integrates into many existing load balancers and proxies like Envoy and nginx, both of which now natively support gRPC directly too.

Re: Data serialization

#75
post #56
post #33

Earlier quoted context omitted.

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or…

That's really weird - why didn't they parse it into a map in the first place, immediately removing duplicate keys? This seems like more of an argument for automatic deserializer generation than it does for adding undue complexity to a serialization format.

Re: Data serialization

#76
post #65
post #52

Earlier quoted context omitted.

I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?

Json is for machines, yaml is for humans, as a general rule. They’re mostly compatible feature-wise.

And if I remember correctly, YAML is a superset of JSON.

Re: Data serialization

#77

I would also recommend CBOR ( http://cbor.io/); one can think of it like a binary form of JSON. It has a few advantages: * datetime objects * binary blobs It is very similar to MsgPack in nature. However, MsgPack, in particular on Python, poorly handles the text/bytes separation, and CBOR is backed by RFC.

BSON is another format that's "binary JSON with blobs and datetimes". It's notably used a lot in MongoDB, but I'm not sure how these three formats compare.

Re: Data serialization

#78
post #56

Earlier quoted context omitted.

Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or…

That's really weird - why didn't they parse it into a map in the first place, immediately removing duplicate keys? This seems like more of an argument for automatic deserializer generation than it does for adding undue complexity to a serialization format.

Even that's not enough to save you in general. JSON deliberately (!) permits duplicate keys. It leaves it up to the implementation to decide how to handle them.

And there are two (sensible) ways to add key/value pairs to a map: either keep the first occurrence of key X, or the last occurrence of key X.

Some JSON libraries pick one way; others pick the other.

Getting the two to interoperate is, as we see, not easy.

It would have been better for JSON to forbid duplicate keys, or to specify a mandatory first-wins or last-wins policy. Then there'd be no room for error.

Re: Data serialization

#80
post #71
post #59

Earlier quoted context omitted.

Both ASN.1 and XML are standardized (with extensive, lengthy and detailed specifications!). The term "S-expression" covers a very broad family of vaguely related not-very-interoperable surface syntaxes.

S-expressions are standardized in both Common Lisp and Scheme. The two standards are not identical, but are mostly compatible, with the CL standard being mostly a superset of the Scheme standard.

Yes, that's exactly what I mean. It's like saying "we use JSON as our file format", but worse.

> "So this program saves its data as 'S-expressions'." > "Oh, cool. Which dialect?" > "Uh, I don't know. It doesn't say in the README. It just says 'S-expressions'."

The answer can be CL, Scheme (lots of variations and dialects even here), the never-finished SPKI Sexps, OCaml sexps, something that the new dev on the team cooked up last Tuesday that vaguely resembles what they learned as "lisp" in college, or something else entirely.

On the other hand, if one were to say "this program uses R4RS S-expressions" (or, presumably, "CL S-expressions", but I haven't read the relevant bits of CLtL), you'd immediately be in a much nicer place than JSON can offer. Not only would you have a well-specified syntax for a reasonably broad range of data types, you'd have a useful equational theory as well. [ETA: Unless you want unicode. Doh. R6RS, maybe.] Ah, the impossible dream.

Post reply on HN