Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

121–130 of 301 posts

Re: Parsing JSON is a Minefield

#121

Post by edoceo »

s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

The article is about how the parsers behave differently. No suggestions of writing from scratch.

The === issue is in JS as well.

Re: Parsing JSON is a Minefield

#122

Post by SomeStupidPoint »

ars wrote: post #108
s_q_b wrote: post #72

Earlier quoted context omitted.

Yeah, the whole idea is a terrible practice, but it is used in industry surprisingly often. People think that PHP's session lock will save them, seeing the potential race condition but not the decode failure. The scenario is typically username-password-parameters passed as three variables via POST. The offending developer parses all three variables up front for simplicity: The session user is created from the POST us…

> and dies on the JSON decode That's the part I don't get. Once the code dies it's done. What exactly can you do now, if no code is even running?

The steps are the following:

1. Parse user.

2. Parse password.

3. Create session object from user and password.

4. Parse JSON. (Crashes here.)

5. Update session and do some other security stuff.

The problem is that they're creating a persistent session object ahead of when the JSON parameters are being decoded, which leaves a (partially initialized, persisting-outside-of-function) session object without having gone through the full authentication flow.

The problem is that they're creating permanent objects ahead of a full validation on the data necessary to actually properly initialize the object.

(I think the OP may have been a little vague because that's probably specific enough to identify the vuln with a scanner and a hunch.)

Re: Parsing JSON is a Minefield

#123

Post by moron4hire »

s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

>> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention

This attitude really pisses me off. Get off your high horse.

Re: Parsing JSON is a Minefield

#124

Post by Animats »

s_q_b wrote: post #53
inimino wrote: post #34

Earlier quoted context omitted.

Parsing HTML is literally orders of magnitude more complex. It is pretty rare to need to parse JSON yourself (what environment doesn't have that available?) but it isn't that difficult. It's a simple language.

Parsing JSON is indeed several orders of magnitude less complex than parsing HTML. In both cases there are excellent parsing libraries, but it would be very unwise to create your own HTML parser for production use, or to use any parser that hasn't seen some serious scrutiny.

JSON at least has the concept of "invalid JSON". That's a big step forward. A JSON parser, like an XML parser, can say "Syntax error - rejected." There's no such thing as "invalid HTML". For that reason, parsing HTML is a huge pain.

As someone who has a web crawler, I'm painfully aware of how much syntactically incorrect HTML is out there. HTML5 has a whole section which standardizes how to parse bad HTML. That's just the syntax needed to parse it into a tree, without considering the semantics at all.

Re: Parsing JSON is a Minefield

#125

Post by vhost- »

You definitely can't rely on it. Just the other day I was given a task to take a request payload from our front end and do some stuff with it on our backend. The payload looked like this: {"thing": [{"values": ["foobar"], "type": "blah blah"}, "some identifier"], "other thing": "some string"}. It's mixing types in arrays which is problematic for most statically types languages.

Tips for Go: Don't use map[string]interface{} and circumvent the type system (I've seen this a lot in production). The fix involves the UnmarshalJSON and MarshalJSON interfaces. This lets you put the data into a structure that's sane and re-encode it back to something the other system expects.

Re: Parsing JSON is a Minefield

#126

Post by firethief »

mi100hael wrote: post #86
Tepix wrote: post #15

Earlier quoted context omitted.

You are not considering a hostile environment like the internet where an attacker will use edge cases to get unforeseen results.

I couldn't really care less if someone POSTs some garbage JSON that results in them getting a 500 response. Better than someone POSTing an XML bomb and affecting other peoples' requests. Please enlighten me if you know of a serialization format with libraries for all common languages that lacks any gotchas or edge cases.

If all you're doing is parsing their JSON for it's own sake, it's just a 500; but that's the boring case. Consider what happens when typical web code is interacting with the JSON parser.

Re: Parsing JSON is a Minefield

#127

Post by Someone »

mi100hael wrote: post #86
Tepix wrote: post #15

Earlier quoted context omitted.

You are not considering a hostile environment like the internet where an attacker will use edge cases to get unforeseen results.

I couldn't really care less if someone POSTs some garbage JSON that results in them getting a 500 response. Better than someone POSTing an XML bomb and affecting other peoples' requests. Please enlighten me if you know of a serialization format with libraries for all common languages that lacks any gotchas or edge cases.

All software has edges, so edge cases are unavoidable.

The best you can do is:

- interpret the spec to the letter.

- for every fragment of a statement you write, consider whether it might conceivably go wrong, and handle those cases (in the simplest matter because 'handling' means writing code, and that code, too, needs to go through this process).

For example, a json parser must be prepared to handle missing values, extremely long keys and values (integers may have thousands of digits, think long about the question whether 64 bits always is enough for storing a string length, etc.), etc.

- if you are truly paranoid, have very stringent security requirements, or expect to be heavily attacked, run the parser in a separate process.

- fuzz your implementation.

Re: Parsing JSON is a Minefield

#128

Post by SloopJon »

mjpa wrote: post #48

Wrote my own JSON parser ( https://github.com/MJPA/SimpleJSON ) a while ago... not sure how it's a minefield unless I'm missing something?

Perhaps you'd be interested to know that your JSONDemo program fails the following tests:

hang:

y_number_huge_exp.json

segfault:

n_structure_100000_opening_arrays.json

n_structure_open_array_object.json

fail:

n_number_then_00.json

n_string_unescaped_tab.json

n_structure_capitalized_True.json

Re: Parsing JSON is a Minefield

#129

Post by austincheney »

Writing parsers is hard and takes some experience, but its not as hard or as impossible as most of these comments make out. JSON is retarted simple to parse, even in the face of certain edge case ambiguities.

I can say this from experience after having written an HTML/XML parser that provides support for various template schemes: Twig, Elm, Handlebars, ERB, Apache Velocity, JSP, Freemarker, and many more. I have written a JavaScript parser that supports React JSX, JSON, TypeScript, C#, Java, and many more things.

In years I have been programming I frequently hear whining like, "its too hard". Don't care. While you are wasting oxygen crying about how hard life is somebody else will roll a solution you will ultimately consume.

Re: Parsing JSON is a Minefield

#130

Post by isidor3 »

petre wrote: post #91
isidor3 wrote: post #29

Earlier quoted context omitted.

Lua was written for exactly this purpose, and I personally enjoy writing with it, so it would be my first choice in most cases.

What purpose? Writing config files? Lua is a programming language (a nice one too). It's code. Code should not be used for config files, nor for data serialization because once you eval it you are executing it.

Yes, Lua was originally written as a language for rich configuration files and has grown out of that into a more fully featured language.

It's also one of the easier languages to sandbox, since you can evaluate user provided code in a custom environment that only contains the functions you deem safe. You can even use the standard debug hooks to set an upper limit on the number of instructions a script can execute to prevent someone from creating an infinite loop in a config file and locking whatever thread is reading the config.

It's not very appropriate as a data serialization format, or as machine written config, but the parent post specifically asked about human written configuration files.

Post reply on HN