Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data serialization

enqueuezero.com

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Data serialization

#81

Post by camgunz »

tonyg wrote: post #56
pitaj wrote: post #33

Earlier quoted context omitted.

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or…

> Do you want remote code execution? Because that's how you get remote code execution.

It's fundamentally not the case that any data serialization format without support for map and set values allows for remote code execution. This CVE was CouchDB's auth code not handling some edge cases in the JSON standard.

Your argument that edge cases like this lead to problems is definitely a good one though. I'm more inclined to say that a lot of these kinds of issues are due to people thinking JSON is a simple, straightforward format when it definitely isn't -- and that's due to mostly two things:

- It fits on a business card!

- You can serialize/deserialize in one line in most implementations

Serialization/Deserialization is something you should always pay careful attention to. Making it a one-liner and advertising it as such was pretty irresponsible.

Re: Data serialization

#82

Post by camgunz »

deathanatos wrote: post #46

I would also recommend CBOR ( http://cbor.io/); one can think of it like a binary form of JSON. It has a few advantages: * datetime objects * binary blobs It is very similar to MsgPack in nature. However, MsgPack, in particular on Python, poorly handles the text/bytes separation, and CBOR is backed by RFC.

CBOR is MessagePack. At least cbor-ruby started with the MessagePack sources. The story is that Carsten took MessagePack, wrote a standard and added some things he wanted, and called it something else.

I wrote [1] a pretty comprehensive (and admittedly biased) critique of the CBOR standard last year.

[1] https://news.ycombinator.com/item?id=14072598

Disclaimer: I wrote and maintain a MessagePack implementation.

Re: Data serialization

#83

Post by teekert »

pletnes wrote: post #65
teekert wrote: post #52

Earlier quoted context omitted.

I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?

Json is for machines, yaml is for humans, as a general rule. They’re mostly compatible feature-wise.

Yaml is well readable by a human but it's much to tightly defined to be "for humans". It's for machines but readable by humans I'd say. Plus it's got sets and you can embed csv's (just 2 things of the top of my head).

Re: Data serialization

#84

Post by strkek »

teekert wrote: post #52
josephg wrote: post #19

Earlier quoted context omitted.

I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see: - Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys) - A standard Date format - Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding. - Arbitrary precision inte…

I'm not an expert, just a bio-data-scientist learning every day... But wouldn't Yaml be what you are looking for?

Obligatory link to the "YAML sucks" repo:

https://github.com/cblp/yaml-sucks

If you ignore the flame-inducing title, it's just a table showing how different implementations parse YAML input in very different ways.

Re: Data serialization

#85

Post by wodenokoto »

linschn wrote: post #18

S-Expressions are missing. Language support is poor (except if you're programming in a Lisp dialect in which case it's built-in), but I do believe they are the best serialization format out there: http://wiki.c2.com/?XmlIsaPoorCopyOfEssExpressions They have a canonical representation https://en.wikipedia.org/wiki/Canonical_S-expressions I swear I have seen a proposal for an efficient binary representation somewhere b…

As far as I understand S-expressions are completely code-as-data, so how do you protect yourself from malicious code execution when loading S-expressions?

Re: Data serialization

#87

Post by soasme »

occamrazor wrote: post #11
udp wrote: post #5

Regarding JSON, this page says: > Performance is not good when dataset is huge. Program usually needs to load all data into memory first. This is just downright false. There are plenty of SAX-style JSON parsers.

Yes. This is a surprising claim in the IP, especially after saying that XML has high performance.

I removed performance piece mostly because it's meaningless discussing performance without in the context of designated implementation and benchmarks.

Above catch is my point so I tweaked the words to `Performance is not good generally when dataset is huge unless you use a library support streaming parsing or writing.`

Re: Data serialization

#88

Post by randyrand »

derefr wrote: post #68
randyrand wrote: post #58

Earlier quoted context omitted.

No need to extend json itself. But the libraries would need to be extended. maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness. embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json. as others said, json already supports arbitrary precision.

> embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json. Those aren't embedded binary blobs, those are string representations of base64-encoded binary blobs. Unless they come out of the JSON decoder as a byte array, they're not "working" as part of JSON; they're another standard on top of JSON. (Also, the GP commenter probably wants them to be transmitted with 0 encoding…

I'm not talking about data URIs. You can write binary data in the same file as the json. Most parsers don't care what comes after the last curly brace.

Re: Data serialization

#89

Post by kazinator »

wodenokoto wrote: post #85
linschn wrote: post #18

S-Expressions are missing. Language support is poor (except if you're programming in a Lisp dialect in which case it's built-in), but I do believe they are the best serialization format out there: http://wiki.c2.com/?XmlIsaPoorCopyOfEssExpressions They have a canonical representation https://en.wikipedia.org/wiki/Canonical_S-expressions I swear I have seen a proposal for an efficient binary representation somewhere b…

As far as I understand S-expressions are completely code-as-data, so how do you protect yourself from malicious code execution when loading S-expressions?

Simply not passing any of these parsed expressions to your eval function.

ANSI Common Lisp presents a pitfall here in that it features read-time evaluation via the #. (hash dot) syntax. For instance #.(+ 2 2) produces the object 4. After seeing #., he reader scans the (+ 2 2) expression, evaluates it immediately, and substitutes the result. When reading untrusted data in Common Lisp, the * read-eval * variable must be set to nil to disable hash-dot.

Lisps that don't have a read-time-eval escape mechanism don't require anything.

Re: Data serialization

#90

Post by guohaochuan »

Our company implemented two rpc protocols(don't ask me why..)

One is based on json, one is modified thrift. The schemaless feature is among the many problems of json, but it's not the most annoying one. That would be binary support.

Imagine we need to make a upload/download storage service with json rpc...So as a rule of thumb, I would consider:

1. Web facing? json would do.

2. RPC? Pick protobuf or thrift or any protocol supporting native types including binary.

Post reply on HN