when parsing human constructed JSON, use JSON5 for the win.
Parsing JSON is a Minefield
111–120 of 301 posts
Re: Parsing JSON is a Minefield
#112Earlier quoted context omitted.
YAML is equally horrible and the spec is an order of magnitude more complex. I wasted half an hour trying to spot an error in the ejabberd yaml config, only to find out something trivial was missing. At least JSON has braces even though it's not suitable for configuration files. By all means choose TOML or something else (even ini or java properties files) instead.
YAML has braces. In fact, it's a super set of JSON. Any YAML parser should be able to parse JSON encoded data with one exception. Block comments. Which is a bastardization of JSON (as mentioned in the article) so block comments shouldn't be a problem in most cases.
> [YAML is] a super set of JSON.
The specification says this, but I was never convinced it was true. Specifically, the spec around escaped unicode characters lacks any mention of surrogates being encoded in two \u sequences, but rather specifies \u as:
> Escaped 16-bit Unicode character.
Which is an unhelpfully not-even-wrong statement. In practice, trying to treat JSON as a subset of YAML results in things not round-tripping, like the following:
In [9]: yaml.load(json.dumps('\N{PILE OF POO}'))
Out[9]: '\ud83d\udca9'
In [14]: _9 == '\N{PILE OF POO}'
Out[14]: False
I've now got surrogates in a decoded Unicode string (why is this even allowed, I don't know); this results in fun behavior like `.encode('utf-8')` raising.
Edit: weird: HN appears to strip pile of poo from comments…
Re: Parsing JSON is a Minefield
#113> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.
> What does HN suggest for configuration files (to be written by a human essentially)? JSON. YAML confuses many people by being whitespace-sensitive; ini files I find too limited.
in Clojure we use EDN (Extensible Data Notation) for config files and a data transfer format. https://github.com/edn-format/edn
It is really a pleasure to use compared to JSON and XML. While it may not be as compact as ProtoBuffers, Thrift, or Avro, it is human readable and also valid Clojure code. Libraries are ready available to convert it to JSON.
Re: Parsing JSON is a Minefield
#114I did write my own parser, but for a reason: I need it to be able to recover as much data as possible from a damaged, malformed, or incomplete file.
Turns out that a good chunk of these tests are for somewhat malformed, but not impossible to reason about files. Extra commas, unescaped characters, leading zeroes... I'd rather just accept those kinds of things rather than throw an error in the user's face. It's a big bad world out there, and data is by definition corrupt.
And this is borne out when I plug my parser into this test suite: Many, many yellow results, which is exactly how I want it.
Re: Parsing JSON is a Minefield
#115What ever happened with EDN (pronounced "eden") from the Clojure people? https://clojure.github.io/clojure/clojure.edn-api.html https://github.com/edn-format/edn I always thought that seemed like a nice alternative data format to JSON. Anyone using this it in the wild?
Clojure programmers use it everywhere. I suspect almost nobody else does though.
Re: Parsing JSON is a Minefield
#116Crockford needs to write "JSON, The Good Parts."
Re: Parsing JSON is a Minefield
#117Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…
Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.
> It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.
That, coupled with the fact that it is still so easy to get it wrong and to introduce security issues is exactly what should peak your attention to the seriousness of the subject. Building any parser is fraught with risk, it is super easy to get it subtly and horribly wrong.
Re: Parsing JSON is a Minefield
#118In practice, people use increasingly smaller subsets of JavaScript to transmit data. For example, a common pattern is to transmit (numeric) user IDs as strings so that they don't get mangled by floating-point precision issues with large numbers. You see both Twitter and Facebook APIs do this, for example.
Modern languages should feature a GUID type in the standard library.
Re: Parsing JSON is a Minefield
#119In practice, people use increasingly smaller subsets of JavaScript to transmit data. For example, a common pattern is to transmit (numeric) user IDs as strings so that they don't get mangled by floating-point precision issues with large numbers. You see both Twitter and Facebook APIs do this, for example.
I've read that you should treat IDs as strings anyway because you want to discourage incrementing IDs, adding IDs together etc. Anything else you want to do with integer IDs, e.g. comparing, can be done with sting IDs as well.
Re: Parsing JSON is a Minefield
#120If JSON is comparable to minefield, then I guess XML and ASN.1 are nothing short of nuclear Armageddon in complexity and ones ability to shoot themselves into the leg ;-)