Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

181–190 of 301 posts

Re: Parsing JSON is a Minefield

#181
post #34

Earlier quoted context omitted.

Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.

It is pretty rare to need to parse JSON yourself but it isn't that difficult. In theory, it's not supposed to be "that difficult". But in practice, according to the linked-to article, due to all rot and general clusterfuck-ery in the various competing specifications, apparently it is. Or do you really think you could wrap your head all around those banana peels, and put together a robust, production-ready parser in a…

Depends upon what you're doing.

I wouldn't want my own JSON parser out on the web, but if I needed to get JSON from $known_environment to $my_service, I'd feel safe enough with a parser I wrote.

Re: Parsing JSON is a Minefield

#182
Do you have an explanation anywhere of why each (or any) of the edge cases is supposed to succeed or fail, or why it commonly does what it's not supposed to do?

I realize that's almost as much work as writing each test case in the first place, but even a subset of the test cases having that explanation would be valuable.

Re: Parsing JSON is a Minefield

#183
People tend to screw up the unicode aspects more than the general parsing. And, indeed, the example JSON parser provided checks for a UTF-8 byte order mark, but doesn't validate that the data is valid UTF-8, so it will let through strings that might cause an application problems.

Although there is a commented out method to validate a code point, so I guess he understood that it was an issue.

Re: Parsing JSON is a Minefield

#184

Earlier quoted context omitted.

Funnily enough, as I've been experimenting with Chef and trying to stick to JSON config files where allowed, I was again struck that (a) it's not a good choice for config files (b) it's an OK choice though (c) lots of people are using it anyway (d) nearly everyone that does so (including Chef) allows comments, so in reality are not actually using JSON at all. Point (d) is the important one. I really think we need a s…

JSON with comments? How about { "object":{ "foo":1,"bar":"New Jersey"} , "_comment": "blah blah blah" } A bit hackish, but always worked for me.

Now add a comment for foo and another one for bar. Also try it when your client throws an exception whenever it encounters invalid keys in object. Or if perhaps it blindly persists or tries to perform some logic using that "data". And so on.

Re: Parsing JSON is a Minefield

#185
post #34

Earlier quoted context omitted.

Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.

It is pretty rare to need to parse JSON yourself but it isn't that difficult. In theory, it's not supposed to be "that difficult". But in practice, according to the linked-to article, due to all rot and general clusterfuck-ery in the various competing specifications, apparently it is. Or do you really think you could wrap your head all around those banana peels, and put together a robust, production-ready parser in a…

[deleted]

Re: Parsing JSON is a Minefield

#186
The correct answer to parsing JSON is... don't. We experimented last hackday with building Netflix on TVs without using JSON serialization (Netflix is very heavy on JSON payloads) by packing the bytes by hand to get a sense of how much the "easy to read" abstraction was costing us, and the results were staggering. On low end hardware, performance was visibly better, and data access was lightening fast.

Michael Paulson, a member of the team, just gave a talk about how to use flatbuffers to accomplish the same sort of thing ("JSOFF: A World Without JSON"), linked in this thread: https://news.ycombinator.com/item?id=12799904

Re: Parsing JSON is a Minefield

#187
post #69

Earlier quoted context omitted.

JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.

JSON's number support is a source of problems, because the standard is so informal [1]: JSON is agnostic about numbers. ... JSON instead offers only the representation of numbers that humans use: a sequence of digits This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into…

Exactly. JSON Number ARE NOT ACTUAL NUMBERS. They're really restricted strings (or, as you quote, a syntax for representing numbers).

IMO this wasn't originally a bad thing at all. There are so many different types of numbers, with so many different behaviours (does `1` == `1.0`? Not in statistics class) that trying to work it all out in JSON would have been a fools errand. The problem is that so many JSON parsers are overly aggressive about turning JSON Numbers into something specific (usually floats) that when writing a JSON API we have to assume it will be consumed in this lowest-common-denomitor way.

By the way, I don't think that all JSON numbers are technically both integers and floats. `42` and `42.0` are clearly different according to the spec, IMO. I'd be curious to get your thoughts on this.

Re: Parsing JSON is a Minefield

#188
post #145

Earlier quoted context omitted.

Silent moment for those of us using niche languages to meet production requirements in environments that do not allow third-party code and do not have JSON parsing in the std lib...

If your language doesn't come with JSON in the stdlib, you're really on the cutting edge. Or using a language meant for embedding :)

Doesn't Java lack a JSON parser in the standard library?

Re: Parsing JSON is a Minefield

#189
post #107

Earlier quoted context omitted.

I don't think so. Can you back it up with facts?

Since you haven't demonstrated your premise beyond talking about what you personally prefer, that particular ball is still in your court. The world isn't obliged to accept your unsupported opinions as truth until you're convinced otherwise.

I don't need to demonstrate anything. It is not de facto standard since if it were everybody were using it which is not the case (Google is the best example). Look up what the term means and you will understand.

Re: Parsing JSON is a Minefield

#190
post #34

Earlier quoted context omitted.

Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.

It is pretty rare to need to parse JSON yourself but it isn't that difficult. In theory, it's not supposed to be "that difficult". But in practice, according to the linked-to article, due to all rot and general clusterfuck-ery in the various competing specifications, apparently it is. Or do you really think you could wrap your head all around those banana peels, and put together a robust, production-ready parser in a…

Well that's the difference: it's the discrepancy between handling the data you're handed, and all the data possible. JSON has a lot of edge cases that are very infrequently exercised. Therefore a "robust, production-ready parser" is not usually what's desired by the pragmatist with the deadline. This can inevitably lead to security holes, but it doesn't necessarily. For example, sometimes the inputs are config files curated by your coworkers, or outputs which come from a server under your control and will always be in UTF-8 and will never use floats or integers larger than 2^50.

Taking it the other way, we can also ask "how can you optimize the parser to be as simple as possible, so that everything is well-specified and nobody can eff it up, while still preserving the structure that JSON gives you?" I tried to experiment with that about five years ago and came up with [1], but it shows a nasty cost differential between "human-readable" and "easy to parse." For example, the easiest string type to parse is a netstring, and this means automatic consistent handling of embedded nulls and what-have-you... but when those unreadable characters aren't escaped then you inherently have trouble reading/writing the file with a text editor. Similarly the easiest type of float is just to take the 64 bits and either dump them directly or as hex... but either way you don't have the ability to properly edit them with the text editor. Etc.

But I am finding that the central problem I'm having with JSON and XML is that it's harder to find (and harder to control!) streaming parsers, so one thing I'm thinking about for the future is that formats that I use will probably need to be streaming from the top-level.[2] So if anyone's reading this and designing stuff, probably even more important than making the parser obviously correct is making it obviously streaming.

[1] https://github.com/drostie/bsencode is based on having an easy-to-parse "outer language" of s-expressions, symbols, and netstrings, followed by an interpretation step where e.g. (float 8:01234567) is evaluated to be the corresponding float.

[2] More recently I've had a lot of success in dealing with more-parallel things to have streamability; for example if you remove whitespace from JSON then [date][tab][process-id][tab][json][newline] is a nice sort of TSV that gets really useful for a workflow of "append what you're about to do to the journal, then do it, then append back that it's done" and so forth; when a human technician needs to go back through the logs they have what they need to narrow down (a) when something went wrong, (b) what else was on that process when it was going wrong, (c) what did it do and what was it trying to do? ... you can of course do all this in JSON but then you need a streaming JSON parser, whereas everyone can do the line-buffering of "buffer a line, split the next chunk by newlines, prepend the first line with the buffer, then save the last line to the buffer, then emit the buffered lines and wait on the next chunk."

Post reply on HN