Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

51–60 of 301 posts

Re: Parsing JSON is a Minefield

#51

Post by andrewstuart2 »

mikeash wrote: post #32
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

A parser should never crash on bad input. If it does, that's a serious bug that needs immediate attention, since that's at least a DoS vulnerability and quite likely something that could result in remote code execution. You definitely need to assume that the parser could fail , but that's different. Unless you're using "crash" in some way I'm not familiar with?

From the context, I'm assuming s_q_b simply means that if given malformed input, the parser should communicate the problem via the appropriate error channels for the given language. In Java, throw an exception; in Go, return an error, etc.

But yeah, the article's example of an XCode segfault is a good example of both poor parsing logic and poor isolation of fault domains. If your json processing library can corrupt the whole process then something's wonky.

Re: Parsing JSON is a Minefield

#52

Post by jerf »

mi100hael wrote: post #3

> In conclusion, JSON is not a data format you can rely on blindly. That was definitely not my take-away from the article. More like "JSON is not a data format you can rely on blindly if you are using an esoteric edge-case and/or an alpha-stage parsing library." I haven't ever run into a single JSON issue that wasn't due to my own fat fingers or trying to serialize data that would have been better suited to something…

The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later.

I'm not saying JSON is bad. Personally I think the sloppiness that this post enumerates is part of its success, and part of why it has so much support in so many languages. The more you nail down the semantics, the harder such widespread support gets. Right now you get a nice subset of the JSON spec that works in a huge variety of languages, where the JSON support for a given language usually converts the JSON into something fairly native for the language. This wouldn't be possible if you nailed down the semantics as hard as something like Protobuf does. For instance, a ton of statically-typed languages will let you encode and decode integers into JSON. But JSON doesn't have an integer type. Strict JSON support ought to forbid integers. But it's sloppiness lets us all just sort of ignore that and get on with life.

If you have a need for precision, avoid JSON. And people probably generally need more precision than they realize and probably ought to reach for JSON a bit less often than they do. But on the other hand, the whole thing does mostly work, right? That can't be ignored.

Re: Parsing JSON is a Minefield

#53

Post by s_q_b »

inimino wrote: post #34
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.

Parsing JSON is indeed several orders of magnitude less complex than parsing HTML. In both cases there are excellent parsing libraries, but it would be very unwise to create your own HTML parser for production use, or to use any parser that hasn't seen some serious scrutiny.

Re: Parsing JSON is a Minefield

#54

Post by Confusion »

There was a great article at some point that explained why 'be liberal in what you accept' is a very bad engineering practice in certain circumstances, such as setting a standard, because it causes users to be confused and annoyed when a value accepted by system A is subsequently not accepted by supposedly compatible system B. Leading to pointless discussions about what the spec 'intended' and subtle incompatibility. Anyone know what article I mean?

Re: Parsing JSON is a Minefield

#55

Post by nulagrithom »

lucb1e wrote: post #23
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Been there done that. (The medical attention, I mean.) Worked just fine. The article makes it sound extremely difficult, but 100% of the article is about edge cases that rarely happen with normal encoders and can often be ignored (e.g. who cares if you escape your tab character?). > consider what ha…

> edge cases that rarely happen with normal encoders and can often be ignored

I've found that "edge cases that rarely happen" are typically 90% of my major headaches.

Re: Parsing JSON is a Minefield

#56

Post by mSparks »

pilif wrote: post #19
robert_tweed wrote: post #9

Earlier quoted context omitted.

Funnily enough, as I've been experimenting with Chef and trying to stick to JSON config files where allowed, I was again struck that (a) it's not a good choice for config files (b) it's an OK choice though (c) lots of people are using it anyway (d) nearly everyone that does so (including Chef) allows comments, so in reality are not actually using JSON at all. Point (d) is the important one. I really think we need a s…

> (b) it's an OK choice though I really think it's not an OK choice. A config file format that doesn't allow comments provides some of the worst possible UX. One of the nice things about config files is that normally they are self-documenting, explaining the meaning of the various directives and providing possible values. Without comments, you have to constantly switch between the documentation and the config file. A…

How does json "not support comments"?

{"comment":"default values for this object"}

Re: Parsing JSON is a Minefield

#57

Post by petre »

indexerror wrote: post #4

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.

YAML is equally horrible and the spec is an order of magnitude more complex. I wasted half an hour trying to spot an error in the ejabberd yaml config, only to find out something trivial was missing. At least JSON has braces even though it's not suitable for configuration files. By all means choose TOML or something else (even ini or java properties files) instead.

Re: Parsing JSON is a Minefield

#58

Post by s_q_b »

andrewstuart2 wrote: post #51
mikeash wrote: post #32

Earlier quoted context omitted.

A parser should never crash on bad input. If it does, that's a serious bug that needs immediate attention, since that's at least a DoS vulnerability and quite likely something that could result in remote code execution. You definitely need to assume that the parser could fail , but that's different. Unless you're using "crash" in some way I'm not familiar with?

From the context, I'm assuming s_q_b simply means that if given malformed input, the parser should communicate the problem via the appropriate error channels for the given language. In Java, throw an exception; in Go, return an error, etc. But yeah, the article's example of an XCode segfault is a good example of both poor parsing logic and poor isolation of fault domains. If your json processing library can corrupt t…

Yes, exactly this.

What I see a lot is initializing an object from a JSON parser which, when it receives malformed input, either halts execution outright or returns an empty object. In both cases you just want to make sure to handle the error appropriately per-language.

Again, the main offender here is PHP, which makes this issue surprisingly easy to get wrong. (And add "==" to your lint checks! 99% of the time coders mean "===", and this can lead to surprisingly severe security issues.)

Re: Parsing JSON is a Minefield

#59

Post by valarauca1 »

mjpa wrote: post #48

Wrote my own JSON parser ( https://github.com/MJPA/SimpleJSON ) a while ago... not sure how it's a minefield unless I'm missing something?

Did you read the post?

The JSON standard(s) are very simple. But in practice this simplicity leads to a lot of edge cases. Very deeply nested structures, numbers that run on to infinity. The point of this post is testing various parsers against these malicious structures.

See this image: http://seriot.ch/json/pruned_results.png

Re: Parsing JSON is a Minefield

#60

Post by DonHopkins »

lucb1e wrote: post #44
indexerror wrote: post #4

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.

> What does HN suggest for configuration files (to be written by a human essentially)? JSON. YAML confuses many people by being whitespace-sensitive; ini files I find too limited.

A general rule of thumb: Never use yet another non-markup language designed by people who claimed to be designing yet another markup language from the very outset, then after somebody awkwardly pointed out that what they'd designed wasn't actually a markup language, they invent a backronym to contradict that embarrassing historical fact.

It just makes me wonder what the hell they thought they were doing all that time... It's like designing something called YACC, and ending up with an interpreter interpreter!

https://en.wikipedia.org/wiki/YAML

>Originally YAML was said to mean Yet Another Markup Language, referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain't Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.

Post reply on HN