Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

191–200 of 257 posts

Re: Parsing JSON is a Minefield

#191

Post by dagenix »

eadmund wrote: post #134
mchanson wrote: post #128

Earlier quoted context omitted.

Oh yes RFC process always keeps things from having compatibility issues. I definitely never saw any issues with all those XML based standards like SOAP or XSLT.

A lot of that is because XML is objectively insane: it's a monumentally over-specified version of something that a sane community would have sketched out on the back of a cocktail napkin. XML is S-expressions done wrong. It's a massive amount of ceremony & boilerplate, IMHO due to the pain of dealing with dynamic data in static languages. It's basically the Java of data-transfer languages. And it shouldn't even be us…

"objectively insane", "over-specified". I'm not sure what "objectively sane" is, but, I guess something specified that way would be a good thing - provided its not "over-specified" since that would be a bad thing, I guess.

Re: Parsing JSON is a Minefield

#192

Post by dagenix »

jstimpfle wrote: post #7
eponeponepon wrote: post #4

This is the one thing that the JSON-against-XML holy warriors need to understand properly. Yes, JSON's less verbose; yes, it's just "plain text" (in as much as there is such a thing); yes, XML makes you put closing tags in - but if you need reliable parsing and rock-solid specifications (and it's reasonably likely that you do, even if you think you don't...), then XML, for all its faults, is very likely the better wa…

Xml sort-of died (in many domains) because of its insane complexity and its redundant ways of specifying relations (child relationships vs explicit relationships, tag name data, attributes data, body data), lack of legibility, parser performance (probably an inherent problem due to hierarchical representation?) and other issues like even meaning of whitespace. >50% of the JSON or Xml I've seen would actually be much…

"Insane complexity"? You mean that it actually has a spec instead of the back of a business card with no implementations that agree on what is actually valid?

Re: Parsing JSON is a Minefield

#193

Post by Someone »

hueving wrote: post #184
Someone wrote: post #118

Earlier quoted context omitted.

More likely, yes, but it need not help you here. Let’s say Chuck sends {“command”:”feed”, “command”:”kill”} Alice uses json parser #1. It keeps both “command” entries. Alice next checks the “command” value against a whitelist. Her json library reads the first value, returning the benign “feed”. Alice next serializes the parsed structure and sends it to Bob. The serializer she uses returns the exact string Eve sent. B…

>What would help here is if Alice generated a clean copy of what she thinks she received, and serialized that. I think you just suggested the same thing the OP did.

Rereading it, you may be right. I read “what was parsed” as “as parsed by the json parser”, but chances are the OP meant “by the application layer”.

I didnkt read it that way because I don’t see that often in my job. Programs there typically know just enough about the format to do their job, and that job doesn’t include “watch out for external threats” (and they don’t all just use some common library that _does_ know the ins and outs of the format because they’re written in different languages. Also, we don’t generate libraries for each language (as would be common if the format were XML) because the json culture doesn’t think json schema is a good idea)

Re: Parsing JSON is a Minefield

#194

Post by couchand »

krapp wrote: post #132
guntars wrote: post #57

Earlier quoted context omitted.

I sure hope you don’t just put random user provided blobs in your database, even if they’re validated. Also, how do you validate without parsing? If it’s parsed, might as well serialize again when saving to the DB.

If it's parsed, why even store in in a database as JSON at all? If you don't do that... then multiple possible JSON parsers aren't a problem.

One common reason is to provide a flexible table for things that may not have an identical schema. For instance, an event log might have details about each event that differ based on the event type.

Re: Parsing JSON is a Minefield

#195

Post by jandrese »

SamReidHughes wrote: post #171
IshKebab wrote: post #110

Earlier quoted context omitted.

Most of the things they test are true of any text-based format, and many of them are true of any serialisation format. E.g. 100000 opening brackets. You could do the same in XML for example and I expect many parsers would fail.

XML is far worse than JSON, it is known. It's a lot easier to screw up a text based format than a binary format. But it would also be possible to make a better text based format than JSON, mainly by defining down what numerical values are allowed to be represented and exactly how they get parsed, and making it harder to screw up strings. That's where most of the problems are.

I think the reason JSON doesn't take well to strong typing is that it is designed for Javascript. When you pass some Javascript object to the JSON encoder how is it supposed to decide which numeric representation to use? Does it always go for the smallest one that works? The Javascript decoder at the other end is just going to throw away that type information anyway, so all it is good for is verifying the data while you are parsing it. Maybe not a totally useless thing to do but it's a lot of work to get that modest benefit.

Re: Parsing JSON is a Minefield

#196

Post by SamReidHughes »

repsilat wrote: post #186
SamReidHughes wrote: post #167

Earlier quoted context omitted.

As a question of fact, programs put out JSON that gets misparsed by other programs. Some simply parse floating point values differently, or they treat Unicode strings incorrectly, or output them incorrectly. Different parsers have different opinions about what a document represents. This has a real world impact. Accepting invalid or ambiguous or undefined JSON is not an acceptable behavior. It means bugs get swallowe…

> Accepting invalid or ambiguous or undefined JSON is not an acceptable behavior Just to make it explicit (and without inserting any personal judgement into the conversation myself): JSON parsers should reject things like trailing commas after final array elements because it will encourage people to emit trailing commas? Having asked the question (and now explicitly freeing myself to talk values) it's new to me -- a…

Postel's Law or the "Robustness Principle" is an anti-pattern in general.

Accepting trailing commas in JSON isn't as big a deal as having two different opinions about what a valid document is. But you might think a trailing comma could indicate a hand-edited document that's missing an important field or array element.

Re: Parsing JSON is a Minefield

#197

Post by SamReidHughes »

jandrese wrote: post #195
SamReidHughes wrote: post #171

Earlier quoted context omitted.

XML is far worse than JSON, it is known. It's a lot easier to screw up a text based format than a binary format. But it would also be possible to make a better text based format than JSON, mainly by defining down what numerical values are allowed to be represented and exactly how they get parsed, and making it harder to screw up strings. That's where most of the problems are.

I think the reason JSON doesn't take well to strong typing is that it is designed for Javascript. When you pass some Javascript object to the JSON encoder how is it supposed to decide which numeric representation to use? Does it always go for the smallest one that works? The Javascript decoder at the other end is just going to throw away that type information anyway, so all it is good for is verifying the data while…

Making an encoding tailored to JavaScript documents that is a binary format is easy. And I don't think strong typing has anything to do with it. Making it handle Python and Ruby objects at the same time is harder, because they have different opinions about what a string is, what a number can be.

Re: Parsing JSON is a Minefield

#198

Post by erik_seaberg »

_greim_ wrote: post #98
bhldr wrote: post #20

Earlier quoted context omitted.

This is also widely considered a bad idea now. Making liberal consumers allows for sloppy producers. Over time this requires new consumers to conform to these sloppy producers to maintain compatibility. Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market.

> Just look at the clusterfuck that HTML5 has become. Ouch. I feel like this is kind of unfair. XML, HTML1-4, and HTML5 all differ in how they treat Postel's law. XML rejects it at the spec level; if you send garbage to a parser it bails immediately, which is nice. HTML5 embraces Postel's law at the spec level. If you send garbage to an HTML5 parser, there's an agreed-on way to deal with it gracefully. Also nice. The…

HTML5 only provides the "be liberal in what you accept" error handling, they have never seen fit to write a "be conservative in what you send" grammar for authors and validators.

Re: Parsing JSON is a Minefield

#199

Post by djur »

eadmund wrote: post #180
djur wrote: post #165

Earlier quoted context omitted.

The S-expression has cleaner whitespace and field names than the JSON, which makes it harder to make an apples-to-apples comparison. But the biggest problem with that S-expression is that I don't know how to parse it. Is SGML a symbol, identifier, a quoteless string? How do I know when parsing the 'entry' field that what follows is going to be a list of key/value pairs without parsing the whole expression? Is 'see-al…

> Is SGML a symbol, identifier, a quoteless string? It's a sequence of bytes — a string, if you like. > How do I know when parsing the 'entry' field that what follows is going to be a list of key/value pairs without parsing the whole expression? You wouldn't, and as a parser you wouldn't need to. The thing which accepts the parsed lists of byte-sequences would need to know what to do with whatever it's given, but tha…

I hope you understand that those questions were rhetorical -- they're questions that do not need to be asked about the equivalent JSON representation. Questions developers don't have to ask each other about the data they're sending each other.

The canonical S-expression representation solves some of the problems JSON has, true, but the example you provided is not a canonical S-expression. It wouldn't make sense for it to have been, because canonical S-expressions are a binary format and not comparable in this context to JSON or XML.

Application developers voted with their feet for serialization formats with native representations of common data types (strings, numbers, lists, maps, booleans, null). There's a lot of reasons that JSON has supplanted XML, but one of them is that JSON has these types built in and XML does not. A lot of real-world data interchange and storage can make good use of those primitives. Many problems boil down to "how do I pass around a list of key/value records". There is a lot to say for not having to renegotiate that kind of basic detail every time two applications need to communicate.

You can represent S-expressions as JSON strings and arrays. I've done it. It was the best way to represent the data I was trying to store, but that's because the data was already represented as S-expressions. I've never seen anyone else do it, and that doesn't surprise me. For most purposes JSON is used for, it is more useful than S-expressions -- not necessarily more powerful, but more useful.

Re: Parsing JSON is a Minefield

#200

Post by benzoate »

dagenix wrote: post #190
arghwhat wrote: post #116

Earlier quoted context omitted.

It really isn't true for JSON either. If you read the rant, most of it is simply about JSON parsers accepting additional, non-JSON syntaxes. Looking at the matrix, all green, yellow, light blue and dark blue are OK outcomes. Red are crashes (stack overflow with 10000 nested arrays, for example), and dark brown are valid JSON that didn't parse (things like UTF-8 mishandling). The issues aren't really JSON-specific.

So if JSON parsers can't agree on what JSON actually is, that isn't a problem?

I consider it more like behaviour outside of the spec is undefined, and some parsers have bugs.
Post reply on HN