Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

191–200 of 301 posts

Re: Parsing JSON is a Minefield

#191

Post by krg »

inimino wrote: post #69
jerf wrote: post #52

Earlier quoted context omitted.

The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later. I'm not saying JSON is bad. Personally I thin…

JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.

"Perhaps all the numbers we use are safe. Are we actually mangling our numbers? Probably not...but will we know? Will anything blow up, or will our application be silently, subtly wrong?

I suspect that when this problem does occur, it goes undetected for longer than it should. In the remainder of this article, we examine potential improvements to our handling of long."

https://www.techempower.com/blog/2016/07/05/mangling-json-nu...

Re: Parsing JSON is a Minefield

#192

Post by mort96 »

RangerScience wrote: post #180
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

I had a need to write my own JSON parser for C#, although that was mostly because I hated the data structures the existing C# parsers produced. I had the advantage that I only needed to use it for rapid prototype projects, and that I could count on all of the data from a single source being the same "shape" (only the scalar values changed, never the keys or objects). Not following the RFC helped greatly, as I just dg…

Wouldn't it have been easier to use the C# JSON parser, and then later walk the tree it creates and convert it into a more sane data structure that way?

Re: Parsing JSON is a Minefield

#193

Post by mort96 »

ars wrote: post #108
s_q_b wrote: post #72

Earlier quoted context omitted.

Yeah, the whole idea is a terrible practice, but it is used in industry surprisingly often. People think that PHP's session lock will save them, seeing the potential race condition but not the decode failure. The scenario is typically username-password-parameters passed as three variables via POST. The offending developer parses all three variables up front for simplicity: The session user is created from the POST us…

> and dies on the JSON decode That's the part I don't get. Once the code dies it's done. What exactly can you do now, if no code is even running?

Remember, it's PHP. Code will be running again on the next request. Only the process (or thread or whatever) handling the request with the invalid JSON crashes, and it only crashes after producing a persistent session; the next request, a different process/thread/whatever will run code, but the session from the previous request still exists.

Re: Parsing JSON is a Minefield

#194

Post by DonHopkins »

mSparks wrote: post #77
pilif wrote: post #63

Earlier quoted context omitted.

Since when is in-band signalling a good idea? What if one of your configuration keys is named "comment"?

{"notes to self":["Don't edit config files by hand","use a decent hierarchy"]} //Http://jsoneditoronline.org

You could even write comments as a linear RSS feed of nested OPML outlines, by converting all that XML to JSON.

http://convertjson.com/xml-to-json.htm

Re: Parsing JSON is a Minefield

#195

Post by clock_tower »

detaro wrote: post #106
marxidad wrote: post #85

Earlier quoted context omitted.

Exceptions should only be used for exceptional cases. For a parser, bad input should be expected.

And in many languages, they are the common way to communicate that a function can not return the data that is expected of it. Invalid input data means the parser can't produce the equivalent data structure -> exception. If the parser has some kind of partial parsing, a way to recover from errors or you are using a language in which returning explicit errors is the more common idiom, then you probably shouldn't throw…

Throwing an exception when parsing fails, sounds like a case of exception-handling as flow of control: a bad thing even when commonly done, having a lot in common with GOTO statements. (See http://softwareengineering.stackexchange.com/questions/18922... , and Ward's Wiki when it comes back up.)

Re: Parsing JSON is a Minefield

#196

Post by atombender »

seagreen wrote: post #187
atombender wrote: post #157

Earlier quoted context omitted.

JSON's number support is a source of problems, because the standard is so informal [1]: JSON is agnostic about numbers. ... JSON instead offers only the representation of numbers that humans use: a sequence of digits This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into…

Exactly. JSON Number ARE NOT ACTUAL NUMBERS. They're really restricted strings (or, as you quote, a syntax for representing numbers). IMO this wasn't originally a bad thing at all. There are so many different types of numbers, with so many different behaviours (does `1` == `1.0`? Not in statistics class) that trying to work it all out in JSON would have been a fools errand. The problem is that so many JSON parsers ar…

The spec is too vague; it's "agnostic". A number can include a fractional part, but it doesn't say if that means that fractionless numbers are to be treated as integers. It leaves that to the implementation, which is a terrible idea for an interchange format.

Ruby, IMHO, errs on the side of consistency (fractionless numbers become Fixnum, so encode(decode("1")) => "1"), whereas Go goes the opposite way (all numbers become float64, so encode(decode("1")) => "1.0", unless you enable the magic json.Number type).

JavaScript is an interesting scenario, because JS doesn't even have integer numbers, which means it doesn't have any choice in the matter. Interestingly, Node.js/V8 truncates the fraction if it can: JSON.dump(JSON.parse("1.0")) => "1".

It's a mess.

Re: Parsing JSON is a Minefield

#197

Post by Someone »

gmfawcett wrote: post #138
Someone wrote: post #127

Earlier quoted context omitted.

All software has edges, so edge cases are unavoidable. The best you can do is: - interpret the spec to the letter. - for every fragment of a statement you write, consider whether it might conceivably go wrong, and handle those cases (in the simplest matter because 'handling' means writing code, and that code, too, needs to go through this process). For example, a json parser must be prepared to handle missing values,…

> think long about the question whether 64 bits always is enough for storing a string length, etc.), etc. I'm struggling to think of any realistic scenario where this isn't true!

So do I, but you should still consciously decide whether to add an overflow check.

Let's do a quick estimate: one can read in the order of 2^24 bytes/second from disk. A day has on the order of 2^16 seconds, so that's 2^40 bytes/day.

=> You will need 2^24 days to read 2^64 bytes. I think that's around 50k years. That an attacker will try to generate a buffer overflow this way is a risk I would take, even if I thought the hardware had room to store that string.

The only way I can foresee a real risk is when an optimizer can optimize away the computation of a string whose length it is asked to compute by an attacker.

That's still very much far-fetched, and if no string gets allocated it's hard to see how it could become a security issue, but it could be a reason to be extra careful, for example when providing online access to a C++ compiler, with its template metaprogramming capabilities.

Re: Parsing JSON is a Minefield

#198

Post by SFJulie »

By sheer randomness I was having the thought about it today: I made some code to highlight where the stdlib json module sees the mistakes in JSON decoding in the python stdlib.

I used the exception with string "blabal at line x, col y, char(c - d)" to actually highlight (ANSI colors) WHERE the mistake were.

https://gist.github.com/jul/406da833d99e545085dac2f368a3b850...

I played a tad with it, and the highlighted area for missing separators, unfinished string, lack of identifier were making no sense. I thought I was having a bug. Checked and re-checked. But, No.

I made this tool because, whatever the linters are I was always wondering why I was not able to edit or validate json (especially one generated by tools coded by idiots) easily.

I thought I was stupid thinking json were complex.

Thanks.

Re: Parsing JSON is a Minefield

#199

Post by prodigal_erik »

grondo4 wrote: post #188
oblio wrote: post #145

Earlier quoted context omitted.

If your language doesn't come with JSON in the stdlib, you're really on the cutting edge. Or using a language meant for embedding :)

Doesn't Java lack a JSON parser in the standard library?

The standard Java library includes a JavaScript interpreter, so there has to be a parser for object literals in there somewhere.

Re: Parsing JSON is a Minefield

#200

Post by 77pt77 »

> it won't parse u-escaped invalid codepoints: ["\ud800"]

How is this not expected behaviour?

The string is not well-formed.

Same thing with decent XML parsers. They croak when you give them invalid codepoints.

Post reply on HN