Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data serialization

enqueuezero.com

Post reply on HN
Print view View original

31–40 of 91 posts

Re: Data serialization

#31

Post by scarface74 »

rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I'm assuming you are using a weakly typed language. My schema is defined by my data objects and I use the built in validation attributes in C#. If the request doesn't pass the validation part of the pipeline, my controller never gets the request and a Bad Request message is sent back to the client.

Re: Data serialization

#32

Post by josephg »

dchest wrote: post #25
josephg wrote: post #19

Earlier quoted context omitted.

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

Arbitrary precision integers is a JavaScript implementation detail; JSON standard doesn't specify number precision.

Huh good to know:

  $ node
  > JSON.parse('1231231231231231231231231123123123123')
  1.2312312312312312e+36
I've heard of JSON implementations in other languages making bigger JSON numbers than javascript supports; but I didn't realise JSON.parse would quietly parse them and throw away precision in the process. I wonder what the plan is for JSON serialization support of bigints. I assume there is no plan - Chrome stable 67 supports bigints, but JSON.serialize(2n) throws a TypeError.

Re: Data serialization

#33

Post by pitaj »

josephg wrote: post #19
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

Re: Data serialization

#34

Post by imtringued »

tannhaeuser wrote: post #15

Don't we do CSV/TSV anymore? It's certainly the most basic schema-less data format imaginable, and you can even have field names using the convention that the first row contains field names. It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it. Tab-separated-values are just data fields separated by a distinguished character, with rows separated by anot…

It's complicated enough that you need a library to properly use it and simultaneously simple enough that people think they don't need a library. Then there are the billions of possible variations. Nobody can agree on a common column (is it ',' ';' or '\t'), newline seperator, whether the first row is a header and you can only store data in the first normal form. CSV is so terrible that even excel spreadsheets are a better data exchange format.

Re: Data serialization

#35

Post by Cthulhu_ »

rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I keep reiterating that there was nothing wrong with XML.

Re: Data serialization

#36

Post by tlocke »

josephg wrote: post #19
rollulus wrote: post #3

Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

You might be interested in Zish https://github.com/tlocke/zish which has support for maps, timestamps, binary data and decimal types.

Re: Data serialization

#37

Post by repsilat »

pitaj wrote: post #33
josephg wrote: post #19

Earlier quoted context omitted.

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

Well, unlees you're using something like capnp, where O(1) contains is meaningful when O(n) deserialise is not acceptable.

And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?

Re: Data serialization

#38

Post by tannhaeuser »

jerf wrote: post #30
tannhaeuser wrote: post #15

Don't we do CSV/TSV anymore? It's certainly the most basic schema-less data format imaginable, and you can even have field names using the convention that the first row contains field names. It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it. Tab-separated-values are just data fields separated by a distinguished character, with rows separated by anot…

"It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it.... no API required." Well, it has no type system, not even "string vs. number", no hierarchy of any kind (no objects, no lists, nothing going deeper), and CSV/TSV is actually a meta-specification requiring a correct use of CSV to provide a complete specification (since you can trust literally nothi…

> canonical [way to] lossly serialize the DOM

I'd rather not :) That's what SGML and XML are designed for.

I agree with your points, but I think the pursuit of the one generalized serialization format to rule them all is misguided and scratching an intellectual itch rather solving a practical problem anyone is having.

In my experience, either you want an upfront interface-first design. When you constrain yourself to mainstream techniques, chances are you'll be using XML/XSD, though it's not ideal to describe co-inductive data structures (it's a grammar-based formalism for text after all).

Or you're not working on system boundaries, and your project doesn't benefit from interface-first design (such as in a Web app with a dedicated back-end). Again if you don't feel like introducing non-mainstream formats, you'll be using JSON.

Everything else seems to be prone to falling into an https://xkcd.com/927/ trap at this point.

Sure there are other use cases (binary data, mass data, streaming, whatever). But these are at least as niche as those for TSV/CSV.

Re: Data serialization

#40

Post by repsilat »

mabbo wrote: post #21

One topic not brought up in this is versioning/change management. While I have my criticisms of Json, if I decide to add a new field to my object, most JSON parsers have the advantage that they'll just consider my old data without that field to have a full value there. XML, same thing. How does protobuf or MsgPack handle that? Aren't they both trying to align data by byte number at some deep level? Or do I not unders…

From memory msgpack is semantically very similar to JSON. It unpacks into arrays and strings and hash tables in more or less the same way, though I think hash table keys are more flexible. It does not come with a schema for object structure or anything, the unpack routines are generic (and pretty "branchy" as a result.)
Post reply on HN