Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

181–190 of 257 posts

Re: Parsing JSON is a Minefield

#181

Post by toast0 »

johannes1234321 wrote: post #138
toast0 wrote: post #96

Earlier quoted context omitted.

My experience with json and similar formats is that most of the complexity arrises from using delimited strings instead of length prefixed strings, and the exciting escaping that results. If the strings are character strings instead of byte strings, you get to add an extra layer of character encoding excitement. PHP serialization is better here, everything is type:value or type:length:value, although strings do have…

The PHP serialisation format has many issues, especially since it allows all sorts of PHP data structures to be encoded. This allows defining references and serializing objects using custom routines into arbitrary binary blobs. Also PHP's unserialization can be used to trigger the autoloader as it tries to resolve unloaded classes, which can trigger unsafe routines in those. Certainly no data format for data exchange…

You shouldn't use PHP's unserialize implementation with untrusted sources; but my point was that its format makes it relatively simple to parse vs json or xml where you have to do a lot of work to parse strings. If you're writing your own parser (including a parser for another language), you could decide to only parse basic types (bool, int, float, array); if you're designing your own format, you could take the lesson of length prefixed strings are much easier to use for computers than delimited strings.

Re: Parsing JSON is a Minefield

#182

Post by Annatar »

arghwhat wrote: post #115
SamReidHughes wrote: post #91

Earlier quoted context omitted.

No, this is not true of many reasonable formats. You don't have to make an obtusely nontrivial format to encode the data JSON does.

JSON is fairly trivial. The post is a nonsensical rant about parsers accepting non-JSON compliant documents (as the JSON spec specifically states that parsers may), such as trailing commas. In the large colored matrix, the following colors mean everything is fine: Green, yellow, light blue and deep blue. Red are crashes (things like 10000 nested arrays causing a stack overflow—this is a non-JSON-specific parser bug),…

Actually unless one is doing JavaScript, JSON is extremely difficult to parse correctly. I challenge you to write a simple, understandable JSON parser in Bourne shell or in AWK.

Re: Parsing JSON is a Minefield

#183

Post by gumby »

bastawhiz wrote: post #46
gumby wrote: post #44

We had a nasty liberal-in-what-you-accept JSON problem: using JSON to communicate between services written in various languages (Python, Java, Javascript, C++): the python client was simply writing maps out which were automatically serialized into something almost JSON: {'label': 123} (using ' to delimit the label strings, not "). The Javascript JSON parser would silently accept this, as would some of the Java librar…

JSON is mostly a strict subset of Python. That's not unexpected, but it makes me question how something like this actually happened. Your bug likely resulted from someone doing a `str(obj)` instead of `json.dumps(obj)`. Hardly the fault of JSON for being very similar to Python's default string serialization.

That is precisely what they did, yes, and some of the client code (e.g. JavaScript) DWIMed it (accepted it)

Re: Parsing JSON is a Minefield

#184

Post by hueving »

Someone wrote: post #118
toast0 wrote: post #99

Earlier quoted context omitted.

Because experience has shown us that today's parsers don't detect tomorrow's 0-day parsing bugs; but serializing a clean version of what was parsed is more likely to be safe (see lots of jpeg, mpeg, etc exploits)

More likely, yes, but it need not help you here. Let’s say Chuck sends {“command”:”feed”, “command”:”kill”} Alice uses json parser #1. It keeps both “command” entries. Alice next checks the “command” value against a whitelist. Her json library reads the first value, returning the benign “feed”. Alice next serializes the parsed structure and sends it to Bob. The serializer she uses returns the exact string Eve sent. B…

>What would help here is if Alice generated a clean copy of what she thinks she received, and serialized that.

I think you just suggested the same thing the OP did.

Re: Parsing JSON is a Minefield

#185

Post by Boulth »

taeric wrote: post #84
Boulth wrote: post #77

Earlier quoted context omitted.

Consumer is interested in fulfilling their need so they will fix their request so that it gets processed.

Or they will pick a service that works with them. This is literally how Google won most of their market share. Sure, there used to be a bit of syntax on the search, but Google always had a single input field and did not barf requests back to users because they put a field in the wrong input.

This is about protocols and data formats not user input text box (the original post mentioned JSON). It's interesting that you bring Google that lead efforts to replace more "liberal" HTTP1.1 with binary, strict HTTP2.

Re: Parsing JSON is a Minefield

#186

Post by repsilat »

SamReidHughes wrote: post #167
arghwhat wrote: post #115

Earlier quoted context omitted.

JSON is fairly trivial. The post is a nonsensical rant about parsers accepting non-JSON compliant documents (as the JSON spec specifically states that parsers may), such as trailing commas. In the large colored matrix, the following colors mean everything is fine: Green, yellow, light blue and deep blue. Red are crashes (things like 10000 nested arrays causing a stack overflow—this is a non-JSON-specific parser bug),…

As a question of fact, programs put out JSON that gets misparsed by other programs. Some simply parse floating point values differently, or they treat Unicode strings incorrectly, or output them incorrectly. Different parsers have different opinions about what a document represents. This has a real world impact. Accepting invalid or ambiguous or undefined JSON is not an acceptable behavior. It means bugs get swallowe…

> Accepting invalid or ambiguous or undefined JSON is not an acceptable behavior

Just to make it explicit (and without inserting any personal judgement into the conversation myself): JSON parsers should reject things like trailing commas after final array elements because it will encourage people to emit trailing commas?

Having asked the question (and now explicitly freeing myself to talk values) it's new to me -- a solid and rare objection to the Robustness Principle. Maybe common enough in these sorts of discussions, though? Anyway, partial as I might be to trailing commas, I do quite like the "JSON's universality shall not be compromised" argument.

Re: Parsing JSON is a Minefield

#187

Post by repsilat »

Annatar wrote: post #182
arghwhat wrote: post #115

Earlier quoted context omitted.

JSON is fairly trivial. The post is a nonsensical rant about parsers accepting non-JSON compliant documents (as the JSON spec specifically states that parsers may), such as trailing commas. In the large colored matrix, the following colors mean everything is fine: Green, yellow, light blue and deep blue. Red are crashes (things like 10000 nested arrays causing a stack overflow—this is a non-JSON-specific parser bug),…

Actually unless one is doing JavaScript, JSON is extremely difficult to parse correctly. I challenge you to write a simple, understandable JSON parser in Bourne shell or in AWK.

> JSON is extremely difficult to parse correctly ... in Bourne shell or in AWK.

Sorry for the misquote, but does it get to the heart of your objection?

I'm torn here. On the one hand I want to say "Those are not languages one typically writes parsers in," but that's a really muddled argument:

1. People "parse" things often in bash/awk because they have to -- because bash etc deal in unstructured data.

2. Maybe "reasonable" languages should be trivially parseable so we can do it in Bash (etc).

I'm kinda torn. On the one hand bash is unreasonably effective, on the other I want data types to be opaque so people don't even try to parse them... would love to hear arguments though.

Re: Parsing JSON is a Minefield

#188

Post by amyjess »

__david__ wrote: post #177
borplk wrote: post #21

I think the idea of humans sharing a language with computers is problematic at a fundamental level. (the whole $dataformat "easy to read for humans") It becomes a source of never ending lose-lose compromises where the more points you give to human convenience the more points you take away from machine convenience and vice versa. Then you end up having to "settle" for something in between that is just ambiguous and pr…

You mean like XBM, XPM, SVG, and EPS?

I remember being horrified to discover that XPM is actually a form of C. That just screams injection risk.

Re: Parsing JSON is a Minefield

#189

Post by tannhaeuser »

eadmund wrote: post #137
drawkbox wrote: post #56

Before JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. Simply exchanging data was a project in itself. At least JSON and XML are text based when it comes to data exchange. Back in the day before APIs that needed to exchange data cleanly, without JSON/XML, exchanging data was not only a minefield but one with constant carpet bombing. The fact that…

> Before JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. S-expressions predate both, are simpler to parse than either, are more legible than both and are cheaper than either. Here's a JSON example from http://json.org/example.html : { "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML"…

Just wanted to share that you can technically parse S-expressions using SGML.

[1]: https://web.archive.org/web/19991008044801/http://www.blnz.c...

Re: Parsing JSON is a Minefield

#190

Post by dagenix »

arghwhat wrote: post #116
skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

It really isn't true for JSON either. If you read the rant, most of it is simply about JSON parsers accepting additional, non-JSON syntaxes. Looking at the matrix, all green, yellow, light blue and dark blue are OK outcomes. Red are crashes (stack overflow with 10000 nested arrays, for example), and dark brown are valid JSON that didn't parse (things like UTF-8 mishandling). The issues aren't really JSON-specific.

So if JSON parsers can't agree on what JSON actually is, that isn't a problem?
Post reply on HN