Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

111–120 of 257 posts

Re: Parsing JSON is a Minefield

#111

Post by taeric »

astrobe_ wrote: post #103
taeric wrote: post #34

Earlier quoted context omitted.

Do you have a survey or other citation for it being a bad idea? I get that it enables bad behavior, per see. However, the idea of rejecting a customer/client because they did not form their request perfectly seems rather anti customer. Ideally, you'd both accept and correct. But that is the idea, just reworded.

It goes against safety. "Accept and correct" in the absence of ECC is just delusion if not hubris. The sender could be in a corrupted state and could have sent data it wasn't supposed to send. Or the data could have been corrupted during transfer, accidentally or deliberately. You can't know unless you have a second communication channel (usually an email to the author of the offending piece of software), and what yo…

In the world of signed requests, but flips are less of a concern. If the signature doesn't match, reject the call. Which implies I clearly don't mean accept literally everything. Just work in your confines and try to move the ball forward, if you can. This is especially true if you are near the user. Consider search engines with the "did you mean?" prompts. Not always correct, but a good feature when few results are found.

For system to system, things are obviously a but different. Don't just guess at what was intended. But, ideally, if you take a date in, be like the gnu date utility and try to accept many formats. But be clear in what you will return.

And, typically, have a defined behavior. That could be to crash. Doesn't have to be, though. Context of the system will be the guide.

Re: Parsing JSON is a Minefield

#112

Post by protomikron »

In my opinion most parsers are just too lax. They support some fancy extension at the beginning (like comments, which are not a good idea), get more and more "feature-rich" and then support syntax that is not specified in the standard.

Other parsers now have to lower their "standard" (no pun intended) to compete which leads to more complex edge-cases that we also find in undefined behaviour and compilers.

E.g. if your HTML is broken, it mostly renders somehow in your browser, which is in my opinion bad design - the same is probably true with JSON.

Re: Parsing JSON is a Minefield

#113

Post by protomikron »

gumby wrote: post #44

We had a nasty liberal-in-what-you-accept JSON problem: using JSON to communicate between services written in various languages (Python, Java, Javascript, C++): the python client was simply writing maps out which were automatically serialized into something almost JSON: {'label': 123} (using ' to delimit the label strings, not "). The Javascript JSON parser would silently accept this, as would some of the Java librar…

> almost JSON: {'label': 123}

Almost is not enough. The parser of an API should reject to transform this into an object. It may be valid Python, but I would blame the first parser (see discussion on Postel's law here).

Re: Parsing JSON is a Minefield

#114

Post by snowpanda »

protomikron wrote: post #112

In my opinion most parsers are just too lax. They support some fancy extension at the beginning (like comments, which are not a good idea), get more and more "feature-rich" and then support syntax that is not specified in the standard. Other parsers now have to lower their "standard" (no pun intended) to compete which leads to more complex edge-cases that we also find in undefined behaviour and compilers. E.g. if you…

I totally agree. JSON to me is actually pretty straight forward, it's the parsers that interpret it differently.

Re: Parsing JSON is a Minefield

#115

Post by arghwhat »

SamReidHughes wrote: post #91
skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

No, this is not true of many reasonable formats. You don't have to make an obtusely nontrivial format to encode the data JSON does.

JSON is fairly trivial. The post is a nonsensical rant about parsers accepting non-JSON compliant documents (as the JSON spec specifically states that parsers may), such as trailing commas.

In the large colored matrix, the following colors mean everything is fine: Green, yellow, light blue and deep blue.

Red are crashes (things like 10000 nested arrays causing a stack overflow—this is a non-JSON-specific parser bug), and dark brown are constructs that should have been supported but weren't (things like UTF-8 handling, which is again non-JSON specific parser bugs).

Writing parsers can be tricky, but JSON is certainly not a hard format to parse.

Re: Parsing JSON is a Minefield

#116

Post by arghwhat »

skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

It really isn't true for JSON either. If you read the rant, most of it is simply about JSON parsers accepting additional, non-JSON syntaxes.

Looking at the matrix, all green, yellow, light blue and dark blue are OK outcomes. Red are crashes (stack overflow with 10000 nested arrays, for example), and dark brown are valid JSON that didn't parse (things like UTF-8 mishandling). The issues aren't really JSON-specific.

Re: Parsing JSON is a Minefield

#117

Post by pjscott »

izacus wrote: post #79
skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

How do Protocol Buffers (which I see used quite alot in similar environments as JSON) compare? Anyone has experience in the format?

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to be straightforward -- no escaping, no backtracking, no ambiguity. I believe the grammar is LL(0), which is a nice touch. And because it's not meant to be human-readable, there's no incentive for people to make their parsers deviate from the strict grammar; e.g. there's no protobuf quirk analogous to JSON's parser-dependent handling of trailing commas, because why would anyone bother?

Re: Parsing JSON is a Minefield

#118

Post by Someone »

toast0 wrote: post #99
Someone wrote: post #87

Earlier quoted context omitted.

”If it’s parsed, might as well serialize again when saving to the DB” You didn’t grow up in the 1980’s, I guess :-) Why spend cycles serializing again if you already have that string?

Because experience has shown us that today's parsers don't detect tomorrow's 0-day parsing bugs; but serializing a clean version of what was parsed is more likely to be safe (see lots of jpeg, mpeg, etc exploits)

More likely, yes, but it need not help you here. Let’s say Chuck sends

  {“command”:”feed”, “command”:”kill”}
Alice uses json parser #1. It keeps both “command” entries.

Alice next checks the “command” value against a whitelist. Her json library reads the first value, returning the benign “feed”.

Alice next serializes the parsed structure and sends it to Bob. The serializer she uses returns the exact string Eve sent.

Bob, using a different json parser, parses the json. That parser drops the first “command”, so he gets the equivalent of

  {“command”:”kill”}
Since Bob trusts Alice, he executes that command.

What would help here is if Alice generated a clean copy of what she thinks she received, and serialized that. For more complex APIs, that would mean she has to know the exact API that Bob expects, though. That may mean extra work keeping Alice’s knowledge of the ins and outs of the API up to date als Bob’s API evolves.

Re: Parsing JSON is a Minefield

#119

Post by cup-of-tea »

kcolford wrote: post #13

This brings to mind the old internet motto (someone correct me on the actual source): "be liberal in what you accept, and be conservative in what you send". JSON is pretty clear on what certain things should mean, strings are Unicode plus escape sequences, objects map keys to values, arrays are ordered collections of values, the whole serialized payload should be Unicode, etc. Even those things can be relaxed further…

I learnt that as the principle of robustness.

Re: Parsing JSON is a Minefield

#120

Post by cptn_brittish »

icebraining wrote: post #100
bjoli wrote: post #59

Earlier quoted context omitted.

JSON can be mapped perfectly to s-expressions. So can xml. Isn't JSON more or less (apart from the commas and colons) just sexprs with a simple schema and different styles of brackets?

I'm not saying it can't be mapped; I'm saying it loses semantics in the translation. For example, how do you represent a boolean in a s-exp, such as that anyone with "the s-exp spec" can unambiguously know that's a boolean?

You use p-lists instead so that {"foo":"bar"} becomes ( :foo "bar")
Post reply on HN