Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

291–300 of 301 posts

Re: Parsing JSON is a Minefield

#291

Post by zaphar »

moron4hire wrote: post #123
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

>> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention This attitude really pisses me off. Get off your high horse.

The implicit subtext here is "for production". It's an excellent rule of thumb since it's highly likely that one already exists that is already battle tested. It's better to use that one. It's not a high horse it's just good horse sense.

He's not talking about rolling your own for fun or learning reasons which anyone should feel free to do.

Re: Parsing JSON is a Minefield

#292

Post by rurban »

s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

Oh my, this is pure nonsense.

First, parsing JSON is trivial compared to other parsing tasks. There are no cycles as in YAML or other serializers, it's trivial forward scanning, without any need to tokenize or backtracking.

Second, JSON is one of the simplest formats out there, and due its simplicity also its most secure. It has some quirks and some edge cases are not well-defined. But with those problems you can always check against your local javascript implementation and the spec, just as OP did.

I know very few JSON parsers which actually crash on illegal input. There are some broken ones, but there are much more broken and insecure by default YAML or BSON parsers or language serializers, like pickle, serialize, Storable, ...

Parsing JSON is not a minefield, parsing JSON is trivial.

Takeaway: Favor JSON over any other serialization format, even if there are some ill-defined edgecases, comments are disallowed and the specs are not completely sound. The YAML and XML specs are much worse, their libraries horrible and bloated.

JSON is the only secure by default serializer. It doesn't allow objects nor code, it doesn't allow cyclic data, no external data, it's trivial, it's fast.

Having summarized that, I'm wondering why OP didn't include my JSON parser in his list, Cpanel::JSON::XS, which is the default fast JSON serializer for perl, is the fastest of all those parsers overall, and is the only one which does pass all these tests. Even more than the new one which OP wrote for this overview STJSON. The only remaining Cpanel::JSON::XS problem is to decode BOM of UTF16 and UTF32. Currently it throws an error. But there are not even tests for that. I added some.

Regarding security: https://metacpan.org/pod/Cpanel::JSON::XS#SECURITY-CONSIDERA...

Re: Parsing JSON is a Minefield

#294

Post by differentman »

int_19h wrote: post #254
oblio wrote: post #145

Earlier quoted context omitted.

If your language doesn't come with JSON in the stdlib, you're really on the cutting edge. Or using a language meant for embedding :)

> you're really on the cutting edge. Or the other way around. Think about something like MUMPS.

No, no thank you. Did it for a year. Pretty sure I don't even want to think about doing it again.

Re: Parsing JSON is a Minefield

#295

Post by z3t4 »

gcirino42 wrote: post #284
z3t4 wrote: post #270

Earlier quoted context omitted.

I want to use something like flat buffers in NodeJS for optimizing websocket traffic and implementing a FS database. But I cant find much stuff for it in JavaScript. Do you (de)serialize the flat buffers or use them directly by abstracting get/set for example via Object.defineProperty ?

Google included a complete example of how to (de)serialize data with them, as well as generating the accessor functions for JS. See https://google.github.io/flatbuffers/flatbuffers_guide_use_j... No, I don't try to decode them directly if I can avoid it. We handrolled a C-based byte packer for our honeybadger project only because function access is relatively slow on the interpreter we have on low end TVs, but readin…

On consideration I don't think flat buffers is worth the added complexity. Using basically C structs is sure faster, and suitable for low end devices and clients written in C. But JSON have many advantages.

Re: Parsing JSON is a Minefield

#296

Post by moron4hire »

zaphar wrote: post #291
moron4hire wrote: post #123

Earlier quoted context omitted.

>> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention This attitude really pisses me off. Get off your high horse.

The implicit subtext here is "for production". It's an excellent rule of thumb since it's highly likely that one already exists that is already battle tested. It's better to use that one. It's not a high horse it's just good horse sense. He's not talking about rolling your own for fun or learning reasons which anyone should feel free to do.

Some of us are not such shitty programmers that we can't build critical systems on our own. Crtitcal business use is absolutely the worst time to farm out to a third party.

And quit using war metaphors to feel macho about your work. What we do in no way resembles battle. Neither are most libraries tested to the degree you are trying to suggest. Neither is common usage in several projects a reliable source if testing.

When people say things like "don't reinvent the wheel" or "only for yourself, not for your employer", what they mean is "I know I can't, and who do you think you are thinking you're better than me. Get back in your place."

Re: Parsing JSON is a Minefield

#297

Post by Macha »

dfox wrote: post #263
Volt wrote: post #247

Earlier quoted context omitted.

JS has stringify and parse, so there ought to be a JSON parser somewhere.

One thing that the article mentions is that there are in fact strings that are valid JSON but not valid JS object literals.

https://developer.mozilla.org/en/docs/Web/JavaScript/Referen...

A modern JS implementation should also have a JSON parser

Re: Parsing JSON is a Minefield

#298

Post by zaphar »

moron4hire wrote: post #296
zaphar wrote: post #291

Earlier quoted context omitted.

The implicit subtext here is "for production". It's an excellent rule of thumb since it's highly likely that one already exists that is already battle tested. It's better to use that one. It's not a high horse it's just good horse sense. He's not talking about rolling your own for fun or learning reasons which anyone should feel free to do.

Some of us are not such shitty programmers that we can't build critical systems on our own. Crtitcal business use is absolutely the worst time to farm out to a third party. And quit using war metaphors to feel macho about your work. What we do in no way resembles battle. Neither are most libraries tested to the degree you are trying to suggest. Neither is common usage in several projects a reliable source if testing.…

You seem to be offended when no offense was intended. No where did the OP indicate that this rule applied to Critical business use. JSON parsing is almost never a critical business use case.

And I don't know about you but these days hosting a public facing internet service is increasingly like a battle DDOS, Data Loss, State sponsored APT's. I certainly don't feel macho about my work. I don't author many libraries that are used by a public facing service on the internet so I have nothing to beat my chest about. I do however use libraries that have been written by someone else and had the bugs shaken out by years of continuous use in a hostile environment. It's not a perfect guarantee but it's certainly better than something I wrote this month.

In most cases a library that has been in continuous use for several years on large public facing services will have been much more tested than anything you might write. Not because you aren't skilled enough but because the edge cases are unbounded and you can't think of everything.

Re: Parsing JSON is a Minefield

#299

Post by seagreen »

rodgerd wrote: post #213
seagreen wrote: post #187

Earlier quoted context omitted.

Exactly. JSON Number ARE NOT ACTUAL NUMBERS. They're really restricted strings (or, as you quote, a syntax for representing numbers). IMO this wasn't originally a bad thing at all. There are so many different types of numbers, with so many different behaviours (does `1` == `1.0`? Not in statistics class) that trying to work it all out in JSON would have been a fools errand. The problem is that so many JSON parsers ar…

> JSON Number ARE NOT ACTUAL NUMBERS. Well, no. They're numbers as humans think about them. (Cue the saying about programming being a job where you hate yourself for not thinking enough like a computer)

Almost two weeks late, but I thought about this more and I totally agree with you.

I should have said "JSON number's aren't machine numbers" (e.g. float, double, integer). But serialized, human-readable numbers are still numbers, so I was wrong.

Re: Parsing JSON is a Minefield

#300

Post by benibela »

mikeash wrote: post #32
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

A parser should never crash on bad input. If it does, that's a serious bug that needs immediate attention, since that's at least a DoS vulnerability and quite likely something that could result in remote code execution. You definitely need to assume that the parser could fail , but that's different. Unless you're using "crash" in some way I'm not familiar with?

It can always crash when there is not enough memory.

Especially on the stack.

On of the test cases has 100000 [ and my own JSON parser crashed on them, because it parsed recursively and the stack overflowed. So now I have fixed it to parse iteratively.

It still crashes.

Big surprise. Turns out the array destructor is also recursive, because it deletes every element in the array. So the array destructor crashes at some random point, after the parser has worked correctly.

Post reply on HN