Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

231–240 of 301 posts

Re: Parsing JSON is a Minefield

#231

Post by umbrai_nation »

clock_tower wrote: post #214
nradov wrote: post #203

Earlier quoted context omitted.

But exception handling is flow control, by its very nature. So it's clearly a gray area and the right thing to do depends on the common idioms of the language you're using, the expected frequency of parsing failures, and (possibly) runtime performance concerns. In Java for example, the XML parser built into the standard library does throw exceptions for certain types of invalid input.

But see Djikstra on "GOTO statement considered harmful" ( http://david.tribble.com/text/goto.html ). The problem is unstructured control flow, which both GOTOs and exceptions-as-control-flow give you; at least in what I was taught (early-2000s CS degree focused on C++), unstructured flow of control is only acceptable when it's a panic button to quit the program (or a major area of processing). It sounds like the Web…

Exception flow control is far more structured than goto. An unexpected goto can drop you anywhere in memory, with no idea how you go there or how to get back. Exceptions cause a clearly-defined chain of events and you always know exactly where you'll end up (passing through all the finally blocks until you hit the closest catch block).

Re: Parsing JSON is a Minefield

#232

Post by mSparks »

DonHopkins wrote: post #194
mSparks wrote: post #77

Earlier quoted context omitted.

{"notes to self":["Don't edit config files by hand","use a decent hierarchy"]} //Http://jsoneditoronline.org

You could even write comments as a linear RSS feed of nested OPML outlines, by converting all that XML to JSON. http://convertjson.com/xml-to-json.htm

Yep, I went through the process of replacing all our XML objects into json ones about 6 years ago now. Smaller files and much easier to read, manipulate store and transfer.

And while I personally quite liked XSLT, javascript is a much more flexible and reliable option.

Re: Parsing JSON is a Minefield

#233

Post by mSparks »

damncabbage wrote: post #217
mSparks wrote: post #149

Earlier quoted context omitted.

Like where? Who for? For what purpose? I cant think of a single example that this would be useful.

A list of items in an array. Some items in that list are of particular note. (Say you have 15 items in an array. For whatever reason, such as you not being in control of the expected input structure of whatever you're giving this JSON to, you have them grouped with comments at the top of each block.)

Why would you want to do that in what is basically a human readable binary file and not in a readme?

You seemed to stop at the for who and for what purpose.

I can 'just' about see the case in something like nodeJS package.json files (that is the least of nodeJS's problems, but that's a whole other conversation). But a readme is a so much better option than having to troll through code comments.

Re: Parsing JSON is a Minefield

#234

Post by inimino »

clock_tower wrote: post #214
nradov wrote: post #203

Earlier quoted context omitted.

But exception handling is flow control, by its very nature. So it's clearly a gray area and the right thing to do depends on the common idioms of the language you're using, the expected frequency of parsing failures, and (possibly) runtime performance concerns. In Java for example, the XML parser built into the standard library does throw exceptions for certain types of invalid input.

But see Djikstra on "GOTO statement considered harmful" ( http://david.tribble.com/text/goto.html ). The problem is unstructured control flow, which both GOTOs and exceptions-as-control-flow give you; at least in what I was taught (early-2000s CS degree focused on C++), unstructured flow of control is only acceptable when it's a panic button to quit the program (or a major area of processing). It sounds like the Web…

It isn't unstructured. It is an exception from the control flow of calling and returning from functions, but it has a structure of its own which is often very convenient for writing clean code. This is not at all specific to the Web, by the way.

Re: Parsing JSON is a Minefield

#235

Post by yes_or_gnome »

deathanatos wrote: post #221
yes_or_gnome wrote: post #216

Earlier quoted context omitted.

That's not wrong. In [1]: import json In [2]: json.dumps("\N{PILE OF POO}") Out[2]: '"\\ud83d\\udca9"' Quality detective work there. JSON, being JavaScript [Object Notation], is standardized to UTF-16. What you see here is "PILE OF POO" in UTF-16. YAML specifies that the recommended encoding is UTF-8, but should be able to parse UTF-16 and UTF-32. And, if you want that, then you need to tell YAML to expect UTF-16. Yo…

> What you see here is "PILE OF POO" in UTF-16. Well, yes, but you missed my point: we're not looking at an encoded string object. A `str` (Python's string type) is supposed to represent a Unicode string — " Strings are immutable sequences of Unicode code points "; the underlying encoding is supposed to be transparent, and in fact, it's possible to construct an example (invalid, IMO, like the above example) YAML that…

> In Unicode's lexicon, this string contains surrogate code points, not surrogate code units. This is wrong, and as I stated, it is surprising that Python permits it. Unicode explicitly warns against this behavior:

> Note that this is the raw JSON/YAML, not a Python repr of it. Those slashes are literal slashes.

I just came to these realization over dinner. Apparently, this is the odd behavior is defined by JSON. So, it's not Python's json module at fault because it is, actually, implemented correctly.

https://en.wikipedia.org/wiki/JSON#Data_portability_issues

This whole time I thought the json module had a bug, but now I am wondering if it's another PyYAML bug.

Maybe not. I'll have to read this section a few more times. http://yaml.org/spec/1.2/spec.html#id2770814

Edit: Going back a bit here.

> Specifically, the spec around escaped unicode characters lacks any mention of surrogates being encoded in two \u sequences,

http://yaml.org/spec/1.2/spec.html#id2771184 makes the statement, "All characters mentioned in this specification are Unicode code points. Each such code point is written as one or more bytes depending on the character encoding used. Note that in UTF-16, characters above #xFFFF are written as four bytes, using a surrogate pair.". And, there's numerous mentions for JSON compatibility. So, I suppose, this is an issue PyYAML (and Ruby's YAML, I checked).

I had never seen this issue before. Ruby's JSON module doesn't breakup utf-8 characters into surrogate pairs, nor does any online json parser that I could find via Google.

Re: Parsing JSON is a Minefield

#236

Post by spc476 »

inimino wrote: post #69
jerf wrote: post #52

Earlier quoted context omitted.

The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later. I'm not saying JSON is bad. Personally I thin…

JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.

I like CBOR (http://cbor.io/). It's a standard (RFC-7049), it can encode 64-bit integers, IEEE floating point, text (UTF-8), binary data (anything non-UTF-8), booleans, null and undefined. You have arrays and maps (and maps can have any value as an index, not just strings) and you can semantically tag data as well. The RFC is one of the better RFCs I've read, covering all the details and plenty of encoding examples to test against.

Re: Parsing JSON is a Minefield

#237

Post by umbrai_nation »

SomeStupidPoint wrote: post #122
ars wrote: post #108

Earlier quoted context omitted.

> and dies on the JSON decode That's the part I don't get. Once the code dies it's done. What exactly can you do now, if no code is even running?

The steps are the following: 1. Parse user. 2. Parse password. 3. Create session object from user and password. 4. Parse JSON. (Crashes here.) 5. Update session and do some other security stuff. The problem is that they're creating a persistent session object ahead of when the JSON parameters are being decoded, which leaves a (partially initialized, persisting-outside-of-function) session object without having gone t…

Wait, so it creates a privileged session before verifying the password? That's your problem right there. A crash in the JSON processor (or anywhere else) is a minor blip compared to this godzilla bug of granting access before it's been earned.

Re: Parsing JSON is a Minefield

#238

Post by jjnoakes »

Kiro wrote: post #173
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

Why would you set the session before the password has been evaluated?

Why do bugs exist?

Re: Parsing JSON is a Minefield

#239

Post by inimino »

mSparks wrote: post #233
damncabbage wrote: post #217

Earlier quoted context omitted.

A list of items in an array. Some items in that list are of particular note. (Say you have 15 items in an array. For whatever reason, such as you not being in control of the expected input structure of whatever you're giving this JSON to, you have them grouped with comments at the top of each block.)

Why would you want to do that in what is basically a human readable binary file and not in a readme? You seemed to stop at the for who and for what purpose. I can 'just' about see the case in something like nodeJS package.json files (that is the least of nodeJS's problems, but that's a whole other conversation). But a readme is a so much better option than having to troll through code comments.

The discussion above concerns why JSON isn't a good choice for configuration files. That's exactly because configuration files are not human readable binary files.

Re: Parsing JSON is a Minefield

#240

Post by memoryhole »

dvt wrote: post #209
gcirino42 wrote: post #186

The correct answer to parsing JSON is... don't. We experimented last hackday with building Netflix on TVs without using JSON serialization (Netflix is very heavy on JSON payloads) by packing the bytes by hand to get a sense of how much the "easy to read" abstraction was costing us, and the results were staggering. On low end hardware, performance was visibly better, and data access was lightening fast. Michael Paulso…

Not sure what your point is (or the point of that presentation, for that matter). Of course there are binary serialization formats that are faster than XML or JSON, and of course they're less error-prone. This has been known for about 40 years now. JSON/XML are used precisely because people want a human-readable interchange format. For high-performance uses, consider Google's Protocol Buffers or Boost::serialize. You…

I believe that is the point. Choose the right serialization strategy to fit the job. Most projects default to JSON regardless of how suitable. At some scale that should be revisited since the human-readable / performance trade-off equation can change.
Post reply on HN