Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

161–170 of 257 posts

Re: Parsing JSON is a Minefield

#161

Post by projektfu »

hyperdimension wrote: post #31
rhapsodic wrote: post #29

Earlier quoted context omitted.

> Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market. What do you mean by "enter that market"?

I think they mean needing deep pockets to write a new browser, with all the complexity that modern HTML+JS entails.

The good news is that Firefox and chromium have pretty open licenses so you only need to change what you want. Of course you need to grok it, which isn't trivial. But writing a browser hasn't been easy since... Cello?

Re: Parsing JSON is a Minefield

#162

Post by drawkbox »

hyperpape wrote: post #150
drawkbox wrote: post #133

Earlier quoted context omitted.

> If other people suggesting that, hey, maybe we should actually be able to express a number correctly makes you splutter about "vogons" or whatever, perhaps it is not they who should take a step back. I guess what I am saying is JSON was created for simplicity and needs no updates. XML has already been created and other formats like BSON, YAML etc or create a new one that suits more detailed needs. The sole reason t…

Forget XML. You can argue that JSON is better because it’s simpler, and while I’m conflicted, I know I enjoy working with JSON more. The real question is: with the benefit of hindsight, could you define a better but similarly simple format? Would an alternative to JSON that specified the supported numeric ranges be less simple? Not really. Would it be better? Yes. The current fact that you can try to represent intege…

> The real question is: with the benefit of hindsight, could you define a better but similarly simple format?

I think the only answer to that question is to build it separate from JSON if you think it can be better, if it is truly better it will win in the market. There is no reason to break JSON and add complexity to the parsing/handling. It is 10x harder to implement simplicity than a format that meets all your needs that ultimately adds complexity.

The problem is when people want to add complexities to JSON. There is nothing stopping anyone from adding a new standard that does do that. But I will argue til the end of time that JSON is successful due to simplicity not edge cases.

Everything you mention can be implemented in JSON just as a string with type info, just because you want the actual type in the format might be the problem, it doesn't fit the use case of simplicity over edge cases. Your use case is one of hundreds of thousands people want in JSON.

> But sometimes one tool could have been better without giving up any of what made it useful for its niche.

Famous last words of a standards implementer. JSON wasn't meant to be this broad, it reached broad acceptance largely because for most cases it is sufficient and simplifies data/messaging/exchange of data. There are plenty of other standards to add complexity or build your own. You use JSON and like it because it is simple.

The hardest thing as an engineer/developer is simplifying complex things, JSON is a superstar in that aspect and I'd like to thank Crockford for holding back on demands like yours. Not because your reasons don't hold value, they do, but because it is moving beyond simplicity and soon JSON would be the past because it will have been XML'd.

In my opinion JSON is one of the best simplifications ever invented in programming and led to tons of innovation as well as simplification of the systems that use it.

If people make JSON more complex we need a SON, Simple Object Notation that is locked Crockford JSON and any dev that wants to add complexity to it will forever be put in the bike shedding shed and live a life of yak shaving.

Re: Parsing JSON is a Minefield

#163

Post by status_quo69 »

freshhawk wrote: post #127

It would be great if programmers learned from markdown and json. Here is the lesson: 1. We need something simpler, so I will make a simple solution to this problem 2. Simple should also mean no strict spec, support for versioning or any of those engineer things. All that engineer shit is boring and I can tell myself this laziness is "staying simple" 3. OH SHIT, I was totally right about #1 so this got popular and hav…

I'm not going to disagree that data formats should undergo a process.... but sometimes they do undergo too much process. Most of the EDI standards, for example, are a product of a huge amount of committee that sells a lot of standards and are used extensively yet are almost impossible to write a parser for.

Edit: obviously this isn't the case, since people have actually parsed it, yet it's a non-trivial thing to do and I have yet to see a solution that isn't horribly specific or a solution that ignores something that might be problematic in non-standard solutions (of which there are a million because of the difficulty involved). A huge amount of domain specificity doesn't help either.

Re: Parsing JSON is a Minefield

#164

Post by haberman »

pjscott wrote: post #117
izacus wrote: post #79

Earlier quoted context omitted.

How do Protocol Buffers (which I see used quite alot in similar environments as JSON) compare? Anyone has experience in the format?

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to be straightforward -- no escaping, no backtracking, no ambiguity. I believe the grammar is LL(0), which is a nice touch. And because it's not meant to be human-readable, there's no incentive for people to make their parsers deviate from the strict grammar; e.g…

I'm glad you had a good experience with Protocol Buffers. (I work on the protobuf team at Google). But I would advise a bit of caution. Writing a fully-compliant protobuf implementation is trickier than it looks. I'd recommend running any new parser through the conformance tests; I created these to exercise some of the edge cases.

https://github.com/google/protobuf/tree/master/conformance

Here are some of the quirks of protobuf:

    - non-repeated fields can occur multiple times on
      the wire -- the last value "wins".
    - you have to be able to handle unknown fields, including
      unknown groups that can be nested arbitrarily.
    - repeated numbers have two different wire formats (packed
      and non-packed), you have be able to handle both.
    - when serializing, all signed integers need to be sign-
      extended to 64 bits, to support interop between different
      integer types.
    - you have to bounds-check delimited fields to make sure
      they don't violate the bounds of submessages you are
      already in.
I do think protobuf is a great technology overall. But it has some complexities too; I wouldn't want to oversell its simplicity and have people be unpleasantly surprised when they come across them later. :)

Re: Parsing JSON is a Minefield

#165

Post by djur »

eadmund wrote: post #137
drawkbox wrote: post #56

Before JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. Simply exchanging data was a project in itself. At least JSON and XML are text based when it comes to data exchange. Back in the day before APIs that needed to exchange data cleanly, without JSON/XML, exchanging data was not only a minefield but one with constant carpet bombing. The fact that…

> Before JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. S-expressions predate both, are simpler to parse than either, are more legible than both and are cheaper than either. Here's a JSON example from http://json.org/example.html : { "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML"…

The S-expression has cleaner whitespace and field names than the JSON, which makes it harder to make an apples-to-apples comparison.

But the biggest problem with that S-expression is that I don't know how to parse it. Is SGML a symbol, identifier, a quoteless string? How do I know when parsing the 'entry' field that what follows is going to be a list of key/value pairs without parsing the whole expression? Is 'see-also GML XML' parsed as a list? How do we distinguish between single element lists and scalars? Is it possible to express a list at the top level, like JSON allows? How do you express a boolean, or null?

Of the problems outlined in the OP, S-expressions solve one: there's no question of how to parse trailing commas because there are no trailing commas. They do not solve questions of maximum levels of nesting. They have the same potential pitfalls with whitespace. They have exactly the same problems with parsing strings and numbers. They have the same problem with duplicated keys.

My point here isn't that you can't represent JSON as S-expressions. Clearly you can. My point is that in order to match what JSON can do, you have to create rules for interpreting the S-expressions, and those rules are the hard part. Those rules, in essence, _are_ JSON; once you've written the logic to serialize the various types supported by JSON to and from S-expressions, you've implemented "JSON with parentheses and without commas".

Re: Parsing JSON is a Minefield

#166

Post by lifthrasiir »

eponeponepon wrote: post #4

This is the one thing that the JSON-against-XML holy warriors need to understand properly. Yes, JSON's less verbose; yes, it's just "plain text" (in as much as there is such a thing); yes, XML makes you put closing tags in - but if you need reliable parsing and rock-solid specifications (and it's reasonably likely that you do, even if you think you don't...), then XML, for all its faults, is very likely the better wa…

No. This is not how you should approach this problem.

The main problem with XML in this regard is a lack of proper data model. In JSON you have a single, mostly consistent data model that is `value ::= atom | [value...] | {key:value...}`. As a data format JSON is half-baked and inefficient, but as a data model it is very clear. On the other hands, XML shines when you actually need the semi-structured markup (which would be very mouthful to represent in JSON).

Re: Parsing JSON is a Minefield

#167

Post by SamReidHughes »

arghwhat wrote: post #115
SamReidHughes wrote: post #91

Earlier quoted context omitted.

No, this is not true of many reasonable formats. You don't have to make an obtusely nontrivial format to encode the data JSON does.

JSON is fairly trivial. The post is a nonsensical rant about parsers accepting non-JSON compliant documents (as the JSON spec specifically states that parsers may), such as trailing commas. In the large colored matrix, the following colors mean everything is fine: Green, yellow, light blue and deep blue. Red are crashes (things like 10000 nested arrays causing a stack overflow—this is a non-JSON-specific parser bug),…

As a question of fact, programs put out JSON that gets misparsed by other programs. Some simply parse floating point values differently, or they treat Unicode strings incorrectly, or output them incorrectly. Different parsers have different opinions about what a document represents. This has a real world impact.

Accepting invalid or ambiguous or undefined JSON is not an acceptable behavior. It means bugs get swallowed up and you can't reliably round trip data.

Re: Parsing JSON is a Minefield

#168

Post by bhldr »

_greim_ wrote: post #98
bhldr wrote: post #20

Earlier quoted context omitted.

This is also widely considered a bad idea now. Making liberal consumers allows for sloppy producers. Over time this requires new consumers to conform to these sloppy producers to maintain compatibility. Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market.

> Just look at the clusterfuck that HTML5 has become. Ouch. I feel like this is kind of unfair. XML, HTML1-4, and HTML5 all differ in how they treat Postel's law. XML rejects it at the spec level; if you send garbage to a parser it bails immediately, which is nice. HTML5 embraces Postel's law at the spec level. If you send garbage to an HTML5 parser, there's an agreed-on way to deal with it gracefully. Also nice. The…

Yea this is absolutely what I meant. HTML5's complexity is a symptom of this problem.

I'm a bit worried about the authors taking this overboard and trying to redefine the URL standard with similar complexity.

Re: Parsing JSON is a Minefield

#169

Post by bhldr »

taeric wrote: post #76
treve wrote: post #63

Earlier quoted context omitted.

The problem with this idea is that different consumers might have a different subset of what they accept and correct. If some of those become dominant, produces might start depending on that behavior and it becomes a de facto standard. This is literally what has happened to HTML, but holds true for many other Internet protocols. If you're looking for some external reading, I found at least this: * https://tools.ietf.…

You'll also find few protocol designers designing anything as robust as the old protocols. :) I mean, don't go out of your way to under specify input. But relatively nobody is going back to the heavy schema of xml over simple json. Even if they probably should. I feel this is an anti fragile position. Try not to encourage poor input. But more importantly, be resilient to it. Not dismissive of it.

Robust as the old protocols? The early TCP protocols are very much underspecified. Maybe you have something else in mind.

Re: Parsing JSON is a Minefield

#170

Post by caf »

taeric wrote: post #149
still_grokking wrote: post #125

Earlier quoted context omitted.

> This is akin to grammar police. In life encounters, there is no real place for grammar policing. However, you should try to be grammatically correct. That's because most humans have feelings. But most machines don't. So that's not comparable.

I meant that grammar policing does little to help the exchange of information. Feelings aside.

A lack of enforced language standards does end up producing a language full of difficult-to-learn inconsistencies, though.
Post reply on HN