Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

71–80 of 301 posts

Re: Parsing JSON is a Minefield

#71
post #35
post #32

Earlier quoted context omitted.

A parser should never crash on bad input. If it does, that's a serious bug that needs immediate attention, since that's at least a DoS vulnerability and quite likely something that could result in remote code execution. You definitely need to assume that the parser could fail , but that's different. Unless you're using "crash" in some way I'm not familiar with?

What about raise an exception?

That qualifies as error reporting in most languages using exceptions.

Re: Parsing JSON is a Minefield

#72
post #23
post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Been there done that. (The medical attention, I mean.) Worked just fine. The article makes it sound extremely difficult, but 100% of the article is about edge cases that rarely happen with normal encoders and can often be ignored (e.g. who cares if you escape your tab character?). > consider what ha…

Yeah, the whole idea is a terrible practice, but it is used in industry surprisingly often. People think that PHP's session lock will save them, seeing the potential race condition but not the decode failure.

The scenario is typically username-password-parameters passed as three variables via POST. The offending developer parses all three variables up front for simplicity:

The session user is created from the POST username, the POST password is input sanitized and put in a variable, then the parameters are parsed as JSON. The attacker POSTs a valid user, a valid (but incorrect) password, and malformed JSON.

The login code reads the variables from POST, opens the session, and dies on the JSON decode.

Hint: this bug exists in the wild on a massively deployed framework. I'm working up a responsible disclosure on that one now.

Re: Parsing JSON is a Minefield

#73
post #64

Now the mess that is called JavaScript dates has crept into any system imaginable in the world. I can understand we needed to go for the lowest denominator but Crockford's card really could cram in another line with a date time string format.

I think adding a date type would have probably doubled the complexity of JSON and taken it right out of the sweet spot that has made it popular.

Re: Parsing JSON is a Minefield

#74
Wow! This was a great practical analysis of existing implementations, besides a great technical overview of the spec(s). Thanks for open sourcing the analysis code[1], and for the extended results[2]

[1] https://github.com/nst/JSONTestSuite

[2] http://seriot.ch/json/parsing.html

Re: Parsing JSON is a Minefield

#75
post #73
post #64

Now the mess that is called JavaScript dates has crept into any system imaginable in the world. I can understand we needed to go for the lowest denominator but Crockford's card really could cram in another line with a date time string format.

I think adding a date type would have probably doubled the complexity of JSON and taken it right out of the sweet spot that has made it popular.

This. Datetime is hard, because RFC 3339 is underspecified [1], and TOML wrestled with this a lot.

[1] https://news.ycombinator.com/item?id=12364393#12364805

Re: Parsing JSON is a Minefield

#76

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.

I've used JSON and YAML for a very long time, but whenever I have the option, I'll be using TOML every time.

YAML is _huge_ (did you know all JSON is valid YAML?) and those features can come back to bite you... http://blog.codeclimate.com/blog/2013/01/10/rails-remote-cod...

JSON has a number of annoyances, mostly no comments, no trailing commas, all the stuff this article gets into.

Formats like INI or CSV don't really have a spec, or if they do, most implementations don't seem to follow them.

TOML is a bit weird at first, but it's grown on me quite a bit.

Re: Parsing JSON is a Minefield

#77
post #63
post #56

Earlier quoted context omitted.

How does json "not support comments"? {"comment":"default values for this object"}

Since when is in-band signalling a good idea? What if one of your configuration keys is named "comment"?

{"notes to self":["Don't edit config files by hand","use a decent hierarchy"]}

//Http://jsoneditoronline.org

Re: Parsing JSON is a Minefield

#78
post #58

Earlier quoted context omitted.

From the context, I'm assuming s_q_b simply means that if given malformed input, the parser should communicate the problem via the appropriate error channels for the given language. In Java, throw an exception; in Go, return an error, etc. But yeah, the article's example of an XCode segfault is a good example of both poor parsing logic and poor isolation of fault domains. If your json processing library can corrupt t…

Yes, exactly this. What I see a lot is initializing an object from a JSON parser which, when it receives malformed input, either halts execution outright or returns an empty object. In both cases you just want to make sure to handle the error appropriately per-language. Again, the main offender here is PHP, which makes this issue surprisingly easy to get wrong. (And add "==" to your lint checks! 99% of the time coder…

If you meant that it would throw or return an error on malformed input, I totally agree. You must always handle errors that your calls might return, and especially when those calls are given anything derived from outside input.

Re: Parsing JSON is a Minefield

#79

    JSON is the de facto standard when it comes to (un)serialising and exchanging data in web and mobile programming. 
I disagree. Take protobuf for example. You get schemas, data structures, and a parser in one package which is actually a lot smaller than JSON and compiles to nearly all the commonly used languages. Ever since I've started using it my life became so easier! If you don't want your data to be human readable (which is very common) you should not use JSON as a data interchange format.

Re: Parsing JSON is a Minefield

#80
post #57

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.

YAML is equally horrible and the spec is an order of magnitude more complex. I wasted half an hour trying to spot an error in the ejabberd yaml config, only to find out something trivial was missing. At least JSON has braces even though it's not suitable for configuration files. By all means choose TOML or something else (even ini or java properties files) instead.

What was the trivial missing thing?
Post reply on HN