Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

91–100 of 257 posts

Re: Parsing JSON is a Minefield

#91

Post by SamReidHughes »

skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

No, this is not true of many reasonable formats. You don't have to make an obtusely nontrivial format to encode the data JSON does.

Re: Parsing JSON is a Minefield

#92

Post by Someone »

jstimpfle wrote: post #32
ProblemFactory wrote: post #28

Earlier quoted context omitted.

XML parsers are necessarily over-complicated for structured data, because it is a text markup language, not a nested data structure language. 123 Hello World Road, 12345 , CA is perfectly sensible XML. The address is not a tree structure or a key-value dictionary - it is free text with optional markup for some words. You can use XML to represent nested data structures with lists and dictionaries, but the parsers and…

Yep, the application to text documents is valid in my eyes, as well. Although there are lighter weight and/or more extensible approaches, like TeX. (update, clarificaton: I mean just the markup syntax, not the compuational model)

Lighter-weight? Tex is Turing-complete. You can’t even know whether interpreting it will ever finish, and writing a parser that produces good error messages on invalid input is difficult.

Re: Parsing JSON is a Minefield

#93

Post by keymone »

xfer wrote: post #69
keymone wrote: post #39

i wish there was a chance for EDN[1] to replace JSON. it's a shame the industry defaulted to a subset of javascript as a data notation format considering all it's shortcomings =/ yeah, i get it, "but it has native support in all browsers" is a valid argument, i just wish it wasn't. [1] https://github.com/edn-format/edn

In what regard EDN is "better" than JSON? the point of the post was the RFC specification is not tight and there are corner cases. I don't see any rigor in the given link either..

it is better in regard that:

- there exists single reference implementation, which rules out points like scalars not being valid JSON in some parsers despite being part of the spec

- it is extensible in a manner that never invalidates the syntax for parsers that do not use corresponding extensions (this is huge actually)

- comments are part of the spec, so it essentially replaces both json and yaml

- richer set of primitive types

- commas are whitespace (best feature ever)

the rest should definitely be handled in BNF spec, but the above makes EDN immediately much better than JSON

Re: Parsing JSON is a Minefield

#94

Post by Groxx »

This seems like more of a problem with parsers not following the spec. It's a simple spec, but so strict and restrictive that it's a bit of a pain to give to humans, and small extensions (like comments) are immensely useful for its (ab)uses as configuration "DSL"s. And some edges like that the string format isn't specified - that's (basically) fine, it's not a connection-negotiation protocol.

So JSON parsers tend to implement some weird, unspecified, inconsistent superset of JSON. I haven't encountered one yet that fails to parse valid JSON though. That doesn't seem to imply that parsing JSON is a minefield. Only that parsing human input, nicely, is a minefield. No spec avoids the human ergonomic problem simply.

Re: Parsing JSON is a Minefield

#96

Post by toast0 »

izacus wrote: post #79
skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

How do Protocol Buffers (which I see used quite alot in similar environments as JSON) compare? Anyone has experience in the format?

My experience with json and similar formats is that most of the complexity arrises from using delimited strings instead of length prefixed strings, and the exciting escaping that results. If the strings are character strings instead of byte strings, you get to add an extra layer of character encoding excitement.

PHP serialization is better here, everything is type:value or type:length:value, although strings do have quotes around them, because their byte length is known, internal quotes need not be escaped. You can still have issues with genrating and parsing the human readible numbers properly (floating point is always fun, and integers may have some bit size limit I don't recall), but you don't need to worry about quoting Unicode values properly.

Protocol buffers have clear length indications, so that's easier, but it's not a 'self documenting' format, you need to have the description file to parse an encoded value. The end result is usually many fewer bits though.

Re: Parsing JSON is a Minefield

#98

Post by _greim_ »

bhldr wrote: post #20
0xcde4c3db wrote: post #15

Earlier quoted context omitted.

> "be liberal in what you accept, and be conservative in what you send" This is commonly known as Postel's Law, and comes from one of the TCP RFCs [1]. [1] https://en.wikipedia.org/wiki/Robustness_principle

This is also widely considered a bad idea now. Making liberal consumers allows for sloppy producers. Over time this requires new consumers to conform to these sloppy producers to maintain compatibility. Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market.

> Just look at the clusterfuck that HTML5 has become.

Ouch. I feel like this is kind of unfair. XML, HTML1-4, and HTML5 all differ in how they treat Postel's law. XML rejects it at the spec level; if you send garbage to a parser it bails immediately, which is nice. HTML5 embraces Postel's law at the spec level. If you send garbage to an HTML5 parser, there's an agreed-on way to deal with it gracefully. Also nice. The problem was rather with HTML1-4, which embraced Postel's law promiscuously, at the implementation level. There were specs, but mainstream implementations largely ignored them and all handled garbage input slightly differently. This is what created the afore-mentioned clusterfuck.

Re: Parsing JSON is a Minefield

#99

Post by toast0 »

Someone wrote: post #87
guntars wrote: post #57

Earlier quoted context omitted.

I sure hope you don’t just put random user provided blobs in your database, even if they’re validated. Also, how do you validate without parsing? If it’s parsed, might as well serialize again when saving to the DB.

”If it’s parsed, might as well serialize again when saving to the DB” You didn’t grow up in the 1980’s, I guess :-) Why spend cycles serializing again if you already have that string?

Because experience has shown us that today's parsers don't detect tomorrow's 0-day parsing bugs; but serializing a clean version of what was parsed is more likely to be safe (see lots of jpeg, mpeg, etc exploits)

Re: Parsing JSON is a Minefield

#100

Post by icebraining »

bjoli wrote: post #59
icebraining wrote: post #25

Earlier quoted context omitted.

But that's the point, isn't it? S-expressions are light because they define very little, it's only a tree of undefined blobs of data (atoms). It's even more limited than JSON.

JSON can be mapped perfectly to s-expressions. So can xml. Isn't JSON more or less (apart from the commas and colons) just sexprs with a simple schema and different styles of brackets?

I'm not saying it can't be mapped; I'm saying it loses semantics in the translation. For example, how do you represent a boolean in a s-exp, such as that anyone with "the s-exp spec" can unambiguously know that's a boolean?
Post reply on HN