Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

221–230 of 257 posts

Re: Parsing JSON is a Minefield

#221

Post by Annatar »

arghwhat wrote: post #217
Annatar wrote: post #182

Earlier quoted context omitted.

Actually unless one is doing JavaScript, JSON is extremely difficult to parse correctly. I challenge you to write a simple, understandable JSON parser in Bourne shell or in AWK.

Why in the world would I write a parser in bash or awk, regardless of the format? I certainly have better things to do. It doesn't matter how sensible a format is, those tools are simply not appropriate to write a parser in. I have, however, written a JSON parser before at a previous company in C++ (they didn't want to pull in a library). It wasn't particularly hard. And yes, like any other parser, it accepted additi…

Why in the world would I have to use JavaScript or any object oriented programming language just because the application is poorly thought out and emits JSON? I certainly have better things to do.

It doesn't matter how sensible a format is, those tools are simply not appropriate to write a parser in.

AWK is a language designed for data parsing and processing. That is what it is designed to do.

How did you solve the parsing of arbitrarily nested structures?

Re: Parsing JSON is a Minefield

#222

Post by _pmf_ »

SamReidHughes wrote: post #171
IshKebab wrote: post #110

Earlier quoted context omitted.

Most of the things they test are true of any text-based format, and many of them are true of any serialisation format. E.g. 100000 opening brackets. You could do the same in XML for example and I expect many parsers would fail.

XML is far worse than JSON, it is known. It's a lot easier to screw up a text based format than a binary format. But it would also be possible to make a better text based format than JSON, mainly by defining down what numerical values are allowed to be represented and exactly how they get parsed, and making it harder to screw up strings. That's where most of the problems are.

> It's a lot easier to screw up a text based format than a binary format.

I don't see a single instance where this is true.

Re: Parsing JSON is a Minefield

#223

Post by nuclx »

borplk wrote: post #21

I think the idea of humans sharing a language with computers is problematic at a fundamental level. (the whole $dataformat "easy to read for humans") It becomes a source of never ending lose-lose compromises where the more points you give to human convenience the more points you take away from machine convenience and vice versa. Then you end up having to "settle" for something in between that is just ambiguous and pr…

I would argue, that YAML (which actually is a superset of JSON) comes closer to JSON in being friendly for humans.

Re: Parsing JSON is a Minefield

#224

Post by kalleboo »

delecti wrote: post #152
bastawhiz wrote: post #47

Earlier quoted context omitted.

Indeed! Facebook spends a ridiculous amount of resources creating text summaries of what's in user-provided images. Check out the alt="" tags next time you're scrolling through your feed. Every single image has one.

I just checked what you're talking about. Creepiness aside, that's kinda awesome.

iOS also does the same thing - if you turn on VoiceOver and go into your photo library it will read out what's in the photos

Re: Parsing JSON is a Minefield

#225

Post by jacobparker »

arghwhat wrote: post #213
dagenix wrote: post #190

Earlier quoted context omitted.

So if JSON parsers can't agree on what JSON actually is, that isn't a problem?

JSON parsers all agree what JSON actually is (except for the brown colored fields in the compat matrix, which are actual JSON compat bugs). JSON parsers may also permit various additional variations which are very explicitly not JSON. This means that they may accept a handcrafted, technically invalid JSON document. However, a JSON encoder may never generate a document containing such "extensions", as this would not b…

> As the robustness principle goes [...]

"The Harmful Consequences of the Robustness Principle" https://tools.ietf.org/html/draft-thomson-postel-was-wrong-0...

Re: Parsing JSON is a Minefield

#226

Post by nnq »

Annatar wrote: post #220
repsilat wrote: post #187

Earlier quoted context omitted.

> JSON is extremely difficult to parse correctly ... in Bourne shell or in AWK. Sorry for the misquote, but does it get to the heart of your objection? I'm torn here. On the one hand I want to say "Those are not languages one typically writes parsers in," but that's a really muddled argument: 1. People "parse" things often in bash/awk because they have to -- because bash etc deal in unstructured data. 2. Maybe "reaso…

Shell and AWK programs have fewest dependencies, are extremely light on resources’ consumption and are extremely fast. When someone’s program emits JSON because the author assumed that programs which ingest that data will also be JavaScript programs, that’s a really bad assumption. It would force the consumer of that data to replicate the environment. This goes against core UNIX principles, as discussed at length in…

> It would force the consumer of that data to replicate the environment.

It won't, because JSON is a standard. Imperfect like all standards but practically good enough. And "plain text" just means "an undefined syntax that I have to mostly guess". And nobody "programs" in bash or awk anymore. The "standard scripting languages" for all sane devs are Python or Ruby (and some Perl legacy) and parsing JSON in them is trivial.

The "UNIX philosophy" was a cancerous bad idea anyway and now it's thankfully being eaten alive by its own children, so time to... rejoice?!

EDIT+: Also, if I feel truly lazy/evil (like in "something I'll only in this JS project"), I would use something much much less standard than JSON, like JSON-5 (https://github.com/json5/json5), which will practically truly force all consumers to use JS :P

Re: Parsing JSON is a Minefield

#228

Post by lmm »

freshhawk wrote: post #127

It would be great if programmers learned from markdown and json. Here is the lesson: 1. We need something simpler, so I will make a simple solution to this problem 2. Simple should also mean no strict spec, support for versioning or any of those engineer things. All that engineer shit is boring and I can tell myself this laziness is "staying simple" 3. OH SHIT, I was totally right about #1 so this got popular and hav…

> look at what happened and think about how Gruber and Crawford will be remembered, partly if not mostly, for being "the asshole who screwed up X".

Completely disagree. Gruber and Crockford are remembered as the people who came up with these really good formats, much better than the overdesigned alternatives.

Re: Parsing JSON is a Minefield

#229

Post by jhomedall »

Annatar wrote: post #221
arghwhat wrote: post #217

Earlier quoted context omitted.

Why in the world would I write a parser in bash or awk, regardless of the format? I certainly have better things to do. It doesn't matter how sensible a format is, those tools are simply not appropriate to write a parser in. I have, however, written a JSON parser before at a previous company in C++ (they didn't want to pull in a library). It wasn't particularly hard. And yes, like any other parser, it accepted additi…

Why in the world would I have to use JavaScript or any object oriented programming language just because the application is poorly thought out and emits JSON? I certainly have better things to do. It doesn't matter how sensible a format is, those tools are simply not appropriate to write a parser in. AWK is a language designed for data parsing and processing. That is what it is designed to do. How did you solve the p…

> How did you solve the parsing of arbitrarily nested structures?

Write a recursive descent parser, or use a parser generator.

Or, more realistically, use one of the many libraries available for parsing it (pretty much every language has one at this point).

Re: Parsing JSON is a Minefield

#230

Post by lmm »

djur wrote: post #156
gnud wrote: post #124

Earlier quoted context omitted.

Won't you inherit all the encoding trouble this article goes on about? As well as the problems with integer precision? And get a bunch of new ones with special characters in keys... I'm sure you _could_ specify a nice sexpr format. I'm not sure the specification would be simple though. And just saying "use sexprs" leaves you with all the problems you have when you say "use json".

To put it another way, the biggest problems with JSON aren't with the representation but with the semantics. S-expressions have no inherent semantics beyond those provided by the structure. Take this S-expression: ((:foo ("bar" 1)) and this JSON: { "foo": [ "bar", 1 ] } and you'd think they have the same meaning, but they don't. The JSON decodes as "an object with one key, foo, which maps to an array containing the s…

> How to interpret the atoms :foo, "bar", and 1, as well as the meaning of their relative positions in the S-expression structure, is the actual hard part.

Indeed, which is surely a downside of S-expressions? Anyone reading the JSON will agree that "bar" and 1 are in a linear sequence, and that sequence is somehow labelled "foo". Whereas different readers of the S-expression might not even agree on that much.

Post reply on HN