Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

241–250 of 301 posts

Re: Parsing JSON is a Minefield

#241

Post by mSparks »

inimino wrote: post #239
mSparks wrote: post #233

Earlier quoted context omitted.

Why would you want to do that in what is basically a human readable binary file and not in a readme? You seemed to stop at the for who and for what purpose. I can 'just' about see the case in something like nodeJS package.json files (that is the least of nodeJS's problems, but that's a whole other conversation). But a readme is a so much better option than having to troll through code comments.

The discussion above concerns why JSON isn't a good choice for configuration files. That's exactly because configuration files are not human readable binary files.

TBH. I find package.json files to be much much much much nicer files than configure.sh or settings.txt files, even a configure.py.

And if that isn't the case for you, not like you don't have the choice.

Re: Parsing JSON is a Minefield

#242

Post by userbinator »

moron4hire wrote: post #123
s_q_b wrote: post #14

Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention. Secondly, assume that parsing your input will crash, so catch the error and have your application fail gracefully. This is the number one security issue I encounter in "security audited" PHP. (The second being the "==" vs. "===" debacle that is PHP comparison.) As one example, consider what happens wh…

>> Well, first and most obviously, if you are thinking of rolling your own JSON parser, stop and seek medical attention This attitude really pisses me off. Get off your high horse.

No kidding. If everyone thought that way, the only JSON parser to exist and which everyone uses would be barely working, because everyone would be put off from trying to write something better.

I have actually written an HTML parser, and it was not so difficult because the HTML spec specifies everything down to the level of tokenising individual characters.

Re: Parsing JSON is a Minefield

#244

Post by userbinator »

gcirino42 wrote: post #186

The correct answer to parsing JSON is... don't. We experimented last hackday with building Netflix on TVs without using JSON serialization (Netflix is very heavy on JSON payloads) by packing the bytes by hand to get a sense of how much the "easy to read" abstraction was costing us, and the results were staggering. On low end hardware, performance was visibly better, and data access was lightening fast. Michael Paulso…

Unfortunately one often has to interact with APIs which require JSON, but I completely agree that pure binary formats are far simpler to work with.

Re: Parsing JSON is a Minefield

#245

Post by userbinator »

dvt wrote: post #209
gcirino42 wrote: post #186

The correct answer to parsing JSON is... don't. We experimented last hackday with building Netflix on TVs without using JSON serialization (Netflix is very heavy on JSON payloads) by packing the bytes by hand to get a sense of how much the "easy to read" abstraction was costing us, and the results were staggering. On low end hardware, performance was visibly better, and data access was lightening fast. Michael Paulso…

Not sure what your point is (or the point of that presentation, for that matter). Of course there are binary serialization formats that are faster than XML or JSON, and of course they're less error-prone. This has been known for about 40 years now. JSON/XML are used precisely because people want a human-readable interchange format. For high-performance uses, consider Google's Protocol Buffers or Boost::serialize. You…

From experience, I think the whole "human-readable" idea is a bit overrated. All it means is that the format is entirely/mostly in ASCII. But if you have a hex editor, like all good programmers should, binary formats are not any less human-readable (or writable) nor more difficult to work with; and for some, even a text editor with CP437 or some other distinctive SBCS will suffice after a while. It's somewhat like learning a language; and if you are the one developing the format, it's a language that you create.

Then again, I grew up working with computers at a time when writing entire apps in Asm/machine language was pretty normal as well as other things which would be considered horribly impossible by many developers of the newer generation, and can mentally assemble/disassemble x86 to/from ASCII, so my perspective may be skewed... just a tiny little bit. ;-)

Re: Parsing JSON is a Minefield

#246

Post by KayEss »

mort96 wrote: post #193
ars wrote: post #108

Earlier quoted context omitted.

> and dies on the JSON decode That's the part I don't get. Once the code dies it's done. What exactly can you do now, if no code is even running?

Remember, it's PHP. Code will be running again on the next request. Only the process (or thread or whatever) handling the request with the invalid JSON crashes, and it only crashes after producing a persistent session; the next request, a different process/thread/whatever will run code, but the session from the previous request still exists.

I still don't get it. The browser gets the session cookie even though the request processing crashed? Or does another request manage to pick up the session that was dropped from the crashed request?

I don't do PHP. It just sounds incredible to me that so little isolation seems to exist.

Re: Parsing JSON is a Minefield

#247

Post by Volt »

tree_of_item wrote: post #208
prodigal_erik wrote: post #199

Earlier quoted context omitted.

The standard Java library includes a JavaScript interpreter, so there has to be a parser for object literals in there somewhere.

Object literals aren't JSON, though.

JS has stringify and parse, so there ought to be a JSON parser somewhere.

Re: Parsing JSON is a Minefield

#248

Post by Volt »

userbinator wrote: post #245
dvt wrote: post #209

Earlier quoted context omitted.

Not sure what your point is (or the point of that presentation, for that matter). Of course there are binary serialization formats that are faster than XML or JSON, and of course they're less error-prone. This has been known for about 40 years now. JSON/XML are used precisely because people want a human-readable interchange format. For high-performance uses, consider Google's Protocol Buffers or Boost::serialize. You…

From experience, I think the whole "human-readable" idea is a bit overrated. All it means is that the format is entirely/mostly in ASCII. But if you have a hex editor, like all good programmers should, binary formats are not any less human-readable (or writable) nor more difficult to work with; and for some, even a text editor with CP437 or some other distinctive SBCS will suffice after a while. It's somewhat like le…

But the phrase is "human-readable" and not "programmer-readable".

Re: Parsing JSON is a Minefield

#249

Post by tblair »

clock_tower wrote: post #223
ethbro wrote: post #219

Earlier quoted context omitted.

Quoting Dijkstra from 1968 on goto as argumentum ad authority in 2016 should be considered harmful. He had reasons for saying what he said in the context of the times. Remove that temporal context and the why becomes more important than the what.

His arguments are still sound, I think, and until that post I'd assumed that avoiding unstructured flow of control was still received wisdom. (I've certainly found it a very sound policy; million-plus-line codebases of other people's code, plus surprises, equal misery.) Joel Spolsky had much the same to say: http://www.joelonsoftware.com/items/2003/10/13.html . If I'm going to have to argue about this, I'd rather use…

The article basically makes 2 points, which are:

1. They are invisible in the source code

2. They create too many possible exit points

As a java developer I really don't find 1 to be a problem at all. Lots of people complain about checked exceptions, but they solve this problem. It's very easy to see which functions throw and which don't. I'd even argue that the invisibility that is there is a positive -- I usually don't want to think about the error case, but can easily trace the exceptional path if I do. I find that often functions need to go back up the stack to handle issues, in which case the exception control flow is exactly what I want.

Runtime exceptions, on the other hand, are invisible and can be an issue, but things like running out of memory, or overflowing the stack, well, I really don't want to have to write: public int add(int x, int y) throws OutOfMemoryException, StackOverflowException, ...

For 2, I find that they just create 1 extra exit point, which is back up the stack/to the catch handler. You could certainly abuse them to cause problems, but I personally don't find this to be an issue in practice.

I think that everyone agrees that unstructured flow of control is problematic, but checked exceptions do have structure, even if it's a bit of a loose one.

Re: Parsing JSON is a Minefield

#250

Post by userbinator »

Animats wrote: post #124
s_q_b wrote: post #53

Earlier quoted context omitted.

Parsing JSON is indeed several orders of magnitude less complex than parsing HTML. In both cases there are excellent parsing libraries, but it would be very unwise to create your own HTML parser for production use, or to use any parser that hasn't seen some serious scrutiny.

JSON at least has the concept of "invalid JSON". That's a big step forward. A JSON parser, like an XML parser, can say "Syntax error - rejected." There's no such thing as "invalid HTML". For that reason, parsing HTML is a huge pain. As someone who has a web crawler, I'm painfully aware of how much syntactically incorrect HTML is out there. HTML5 has a whole section which standardizes how to parse bad HTML. That's jus…

There's no such thing as "invalid HTML". For that reason, parsing HTML is a huge pain.

Actually, as someone who has written an HTML parser by following the HTML5 spec, I see it as the opposite: because every string of bytes essentially corresponds to some HTML tree, there are no special "invalid" edge cases to consider and everything is fully specified. That's the best situation, since bugs tend to arise at the edge cases, the boundaries between valid and invalid. But the HTML5 spec has no edge cases: the effect of any byte in any state of the parser has been specified.

HTML5 has a whole section which standardizes how to parse bad HTML.

In some ways, I think it's a bit of a moot point what is "bad HTML" if all parsers that conform to the standard parse it in the same way. The spec does mention certain points as being parse errors, but are they really errors after all, if the behaviour is fully specified (and isn't "give up")? In fact, disregarding whether some states are parse errors actually simplifies the spec greatly because many of the states can be coalesced, and for something like a browser or crawler, it's completely irrelevant whether any of these "parse errors" actually occurred during the parsing. One example that comes to mind is spaces after quoted attribute values; a="b" c="d" and a="b"c="d" are parsed identically, except the latter is supposedly a parse error. Yet both are unambiguous.

Post reply on HN