Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data serialization

enqueuezero.com

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Data serialization

#51

Post by jhpriestley »

Cthulhu_ wrote: post #35
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I keep reiterating that there was nothing wrong with XML.

parsing xml can take exponential space and requires an internet connection!

Re: Data serialization

#52

Post by teekert »

josephg wrote: post #19
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?

Re: Data serialization

#53

Post by tomcatfish »

jhpriestley wrote: post #51
Cthulhu_ wrote: post #35

Earlier quoted context omitted.

I keep reiterating that there was nothing wrong with XML.

parsing xml can take exponential space and requires an internet connection!

>requires an internet connection Please explain this point in particular. I can't seen to imagine what this could possibly mean.

Re: Data serialization

#54

Post by phyrex »

josephg wrote: post #19
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

Do you know of Transit? https://github.com/cognitect/transit-format

Re: Data serialization

#55

Post by [deleted] »

rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

[deleted]

Re: Data serialization

#56

Post by tonyg »

pitaj wrote: post #33
josephg wrote: post #19

Earlier quoted context omitted.

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html

Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or a set. Shortcuts are taken, consumers disagree on the details, and the linked CVE is the result. Being able to reliably convey "this is intended to be a map", or "this is intended to be a set" is crucial for secure and robust interoperability.

Re: Data serialization

#57

Post by vanderZwan »

I wonder why FlatBuffers aren't a more popular option[0]. The format isn't even mentioned in the article. I have never used it (since I my needs were not that advanced), but it seem to combine quite a few amazing qualities of other serialization formats, while adding a few of its own. It even has a version without a schema called "FlexBuffers"[1]. I would expect anyone considering ProtoBuffers and MsgPack to give FlatBuffers a look.

Two five minute lightning talks by Van Oortmerssen, who created the format (another reason to take a closer look IMO), that shows off some of the powerful features:

Lightning Talk: FlatBuffers (2015)

https://www.youtube.com/watch?v=olmL1fUnQAQ

Going Further with FlatBuffers

https://www.youtube.com/watch?v=90ND0yQVYg8

[0] https://google.github.io/flatbuffers/

[1] https://google.github.io/flatbuffers/flexbuffers.html

Re: Data serialization

#58

Post by randyrand »

josephg wrote: post #19
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

No need to extend json itself. But the libraries would need to be extended.

maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.

embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json.

as others said, json already supports arbitrary precision.

Re: Data serialization

#59

Post by tonyg »

zeveb wrote: post #50
teddyh wrote: post #44

Earlier quoted context omitted.

Or ASN.1.

Or S-expressions.

Both ASN.1 and XML are standardized (with extensive, lengthy and detailed specifications!). The term "S-expression" covers a very broad family of vaguely related not-very-interoperable surface syntaxes.

Re: Data serialization

#60

Post by blattimwind »

tomcatfish wrote: post #53
jhpriestley wrote: post #51

Earlier quoted context omitted.

parsing xml can take exponential space and requires an internet connection!

>requires an internet connection Please explain this point in particular. I can't seen to imagine what this could possibly mean.

In the most general sense an XML document can reference schema(s) not locally available; a client wishing to validate against those schemas has to download them.

For most use cases, this isn't relevant, because the application expects a document of schema X and will provide said schema as well.

Post reply on HN