If you're storing tons of numbers you might consider scientific formats like HDF5, ROOT or FITS.
Data serialization
61–70 of 91 posts
Re: Data serialization
#62Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That…
I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…
Another couple of missings:
- NaN/inf floats
- Comments
Re: Data serialization
#63One topic not brought up in this is versioning/change management. While I have my criticisms of Json, if I decide to add a new field to my object, most JSON parsers have the advantage that they'll just consider my old data without that field to have a full value there. XML, same thing. How does protobuf or MsgPack handle that? Aren't they both trying to align data by byte number at some deep level? Or do I not unders…
msgpack & Co. are basically JSON-as-TLV (type-length-value) using a variable-length integer encoding for every integer.
Re: Data serialization
#64Earlier quoted context omitted.
I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…
Arbitrary precision integers is a JavaScript implementation detail; JSON standard doesn't specify number precision.
In theory, that is true. In practice, if the consuming client is javascript, it is really hard to map the numbers into an arbitrary precision number upon decoding (which would come from some library like big.js), rather than a pure double.
Which means, in practice, you are stuck passing numbers as strings (and on the javascript side, pass that string into the arbitrary precision number library)
Re: Data serialization
#65Earlier quoted context omitted.
I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…
I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?
Json is for machines, yaml is for humans, as a general rule. They’re mostly compatible feature-wise.
Re: Data serialization
#66I wonder why FlatBuffers aren't a more popular option[0]. The format isn't even mentioned in the article. I have never used it (since I my needs were not that advanced), but it seem to combine quite a few amazing qualities of other serialization formats, while adding a few of its own. It even has a version without a schema called "FlexBuffers"[1]. I would expect anyone considering ProtoBuffers and MsgPack to give Fla…
I've used it in production, can confirm it is awesome.
Flatbuffers destroys ProtoBuf both in terms of allocations(1 vs N where N is usually huge) and deserialization speed.
You also get explicit control over your data layout in memory when you write the message. So if you know what your read patterns are up-front you can get amazing cache usage even in managed languages.
Re: Data serialization
#67If you're storing tons of numbers you might consider scientific formats like HDF5, ROOT or FITS.
Add netcdf4, which builds upon hdf5.
Re: Data serialization
#68Earlier quoted context omitted.
I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…
No need to extend json itself. But the libraries would need to be extended. maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness. embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json. as others said, json already supports arbitrary precision.
> embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json.
Those aren't embedded binary blobs, those are string representations of base64-encoded binary blobs. Unless they come out of the JSON decoder as a byte array, they're not "working" as part of JSON; they're another standard on top of JSON.
(Also, the GP commenter probably wants them to be transmitted with 0 encoding overhead. Which can't really work while JSON is still JSON. But you can always use an alternative format which handles a superset of JSON's types, and which is binary, e.g. BSON or CBOR.)
> maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.
I think you're fundamentally misunderstanding the thrust of the GP poster's request, here? They don't want to serialize a map in a way that is cheap to deserialize—different languages have different hashtable implementations so there's no way it could really work. What the GP poster (and many other people) want, is just to have something that encodes similarly to existing "object maps" (the ones with curly braces), but with a slight syntactic alteration so that they come out of the decoder as Maps, rather than as Objects.
Re: Data serialization
#69Earlier quoted context omitted.
Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.
Well, unlees you're using something like capnp, where O(1) contains is meaningful when O(n) deserialise is not acceptable. And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?
> You wouldn't say that objects/dicts in JSON are irrelevant, would you?
I kind of would. Coming from Erlang, I don't see a point to there being a whole separate syntax for encoding object/map types. Just encode objects/maps as arrays of pairs (2-Tuples, or just length-2 arrays).
Then, if you want ser-des type fidelity, stick an annotation onto the array (like in YAML) to say what type it should decode to.
E.g., something like:
[1, 2, 3] # array
[['a', 1], ['b', 2], ['c', 3]] # array of pairs
[@object, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to an Object
[@map, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to a Map
Re: Data serialization
#70[deleted]