Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

271–280 of 301 posts

Re: Parsing JSON is a Minefield

#271

Post by mathw »

int_19h wrote: post #252
inimino wrote: post #224

Earlier quoted context omitted.

Writing any code is fraught with risk, but writing a parser in a modern and reasonably safe language is not something to be greatly feared. It's more likely that you'll introduce a security issue in what you do with the JSON immediately after you parse it.

> writing a parser in a modern and reasonably safe language is not something to be greatly feared It ought to be feared, if interoperability is involved. The problem isn't that you might introduce security issues. The problem is usually that you introduce very subtle deviations to the spec that everyone else implemented correctly, and as a result, sometimes your input and/or output do not work with other stuff out th…

Writing a parser for a badly-specified format which is widely used is a terrifying prospect in any language.

Okay so it's more terrifying in C than most other things, but still, it's terrifying. Runaway memory consumption, weird Unicode behaviour etc. etc. etc. It's easy to think you don't have to worry about Unicode because your language's string types will handle it for you - but what do they do if the input contains invalid codepoints? You're writing a parser, you need to know - and possibly override it if that behaviour conflicts with the spec.

Horrible business. Definitely not my favourite job.

Re: Parsing JSON is a Minefield

#272

Post by mathw »

umbrai_nation wrote: post #237
SomeStupidPoint wrote: post #122

Earlier quoted context omitted.

The steps are the following: 1. Parse user. 2. Parse password. 3. Create session object from user and password. 4. Parse JSON. (Crashes here.) 5. Update session and do some other security stuff. The problem is that they're creating a persistent session object ahead of when the JSON parameters are being decoded, which leaves a (partially initialized, persisting-outside-of-function) session object without having gone t…

Wait, so it creates a privileged session before verifying the password? That's your problem right there. A crash in the JSON processor (or anywhere else) is a minor blip compared to this godzilla bug of granting access before it's been earned.

Spot on! But if the JSON parser couldn't crash on malformed input, that kind of whopping mistake would be a lot harder to exploit.

Re: Parsing JSON is a Minefield

#273

Post by yazaddaruvala »

iopq wrote: post #262
mypalmike wrote: post #222

Earlier quoted context omitted.

Exceptions are for flow control. That's their entire purpose. Throwing an exception when parsing fails is a near perfect example of where exceptions produce clarity of code. That is to say, a parse function is usually a pure function that takes a string and returns some sort of object. As a pure function, it "answers a question". A parser answers the question: "What object does this string represent?" When given bad…

In Rust, it would be Option where T is the type of the object you're supposed to get. This means that if you don't parse correctly, it would be None, if you do parse correctly it would be Some(T)

Instead of `Option` it would be more idiomatic to use `Result` or even better `Result`. Where `E` is some way to communicate the error code/type/message back to the caller.

Re: Parsing JSON is a Minefield

#274

Post by inimino »

atombender wrote: post #157
inimino wrote: post #69

Earlier quoted context omitted.

JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.

JSON's number support is a source of problems, because the standard is so informal [1]: JSON is agnostic about numbers. ... JSON instead offers only the representation of numbers that humans use: a sequence of digits This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into…

> You would expect encode(decode(string)) == string

With Unicode string handling I already wouldn't expect that, not to mention whitespace. However, I would expect that dec(enc(dec(string))) === dec(string)

> {"visitorCount": 42.0} is perfectly valid

The problem here is not necessarily that the standard is informal, but that languages differ in the number types they offer. Treating "42.0" as something different from "42" is after all just another convention and not a universal one. In other words, having to coerce something to an integer is more a property of the environment than a problem with JSON.

Re: Parsing JSON is a Minefield

#275

Post by TeeWEE »

s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

Takeaway: Dont parse json. Every json parser is wrong. Json is not a standard.

Parsing everything is NOT a minefield. We have BNF's parser theory etc for that. Lots of languages have clear unambigious definitions... Json clearly not. ITs a disgrace for the software engineering community.

Re: Parsing JSON is a Minefield

#277

Post by burntsushi »

mypalmike wrote: post #266
iopq wrote: post #262

Earlier quoted context omitted.

In Rust, it would be Option where T is the type of the object you're supposed to get. This means that if you don't parse correctly, it would be None, if you do parse correctly it would be Some(T)

Consider this: def load_data(): with open('some_file.json', 'r') as f_in: data = parse_json_stream(f_in) data['timestamp'] = some_date_fn() # Do something with the *definitely-valid* data on the next line. return data def parse_json_stream(io_stream): # Some complex parser... # at some point... if next_char != ']': raise JsonException('Expected "]" at line {}, column {}'. format(line, col)) # More parser code... A be…

Rust nearly has that same benefit. You can wrap `parse_json_stream(f_in)` in `try!(parse_json_stream(f_in))`, and if an error was returned from `parse_json_stream`, then an early return for `load_data` is inserted automatically, thereby propagating the error, similar to raising an exception.

Of course, these approaches are not isomorphic, but in Rust, the cost of explicitly checking an error value is typically very small thanks to algebraic data types, polymorphism and macros.

Re: Parsing JSON is a Minefield

#278

Post by rurban »

_wiv7 wrote: post #10

"NaN and Infinity" Yeah. And I learned this the hard way with the Perl module JSON::XS. It successfully encodes a Perl NaN, but its decoder will choke on that JSON. (Reported it to the maintainer who insists that is consistent with the documentation and wouldn't fix it)

Nobody should use JSON::XS anymore, everybody switched to Cancel::JSON::XS which does have those features, and less bugs.

I just added those great testcases, and found previously unknown bugs. But the python test runner from this repo gives a few false negatives, PR coming soon. It was much easier to write it in perl.

Re: Parsing JSON is a Minefield

#279

Post by valarauca1 »

mjpa wrote: post #96
valarauca1 wrote: post #59

Earlier quoted context omitted.

Did you read the post? The JSON standard(s) are very simple. But in practice this simplicity leads to a lot of edge cases. Very deeply nested structures, numbers that run on to infinity. The point of this post is testing various parsers against these malicious structures. See this image: http://seriot.ch/json/pruned_results.png

"In conclusion, JSON is not a data format you can rely on blindly." - that suggests the format is bad when it's highlighting problems with the parsers. I see it like saying "Plain text is a bad format because notepad bails on large files"

    In conclusion, NOUN1 is not a NOUN2 you can rely on blindly
This is true for everything. I don't see why this is evidence of anything. If you rely on any system blindly you are doing something wrong.

Re: Parsing JSON is a Minefield

#280

Post by paulddraper »

edem wrote: post #189
oldmanjay wrote: post #132

Earlier quoted context omitted.

Since you haven't demonstrated your premise beyond talking about what you personally prefer, that particular ball is still in your court. The world isn't obliged to accept your unsupported opinions as truth until you're convinced otherwise.

I don't need to demonstrate anything. It is not de facto standard since if it were everybody were using it which is not the case (Google is the best example). Look up what the term means and you will understand.

Google uses JSON for their APIs.

https://developers.google.com/drive/v2/reference

Post reply on HN