Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

231–240 of 257 posts

Re: Parsing JSON is a Minefield

#231

Post by ric2b »

projektfu wrote: post #161
hyperdimension wrote: post #31

Earlier quoted context omitted.

I think they mean needing deep pockets to write a new browser, with all the complexity that modern HTML+JS entails.

The good news is that Firefox and chromium have pretty open licenses so you only need to change what you want. Of course you need to grok it, which isn't trivial. But writing a browser hasn't been easy since... Cello?

Forking an existing browser ins't exactly writing a new browser, it's modifying one. That process very likely limits innovation in the space.

Re: Parsing JSON is a Minefield

#232

Post by taeric »

Boulth wrote: post #185
taeric wrote: post #84

Earlier quoted context omitted.

Or they will pick a service that works with them. This is literally how Google won most of their market share. Sure, there used to be a bit of syntax on the search, but Google always had a single input field and did not barf requests back to users because they put a field in the wrong input.

This is about protocols and data formats not user input text box (the original post mentioned JSON). It's interesting that you bring Google that lead efforts to replace more "liberal" HTTP1.1 with binary, strict HTTP2.

My point is often that those that are pushing for stricter formats have good intentions. Strong arguments, even. However, what is required to grow the adoption of something is different than hardening it. And typically hardening something makes it brittle is some way. (Which is super risky at the adoption stage.)

And, of course, most people don't actually understand why they succeeded at something. It is easy to understand failure from a specific cause. It is much more difficult to understand success from a combination of many causes.

Re: Parsing JSON is a Minefield

#233

Post by bloak »

spc476 wrote: post #153
bufferoverflow wrote: post #71

Earlier quoted context omitted.

> anything poorly specified I thought JSON was specified quite clearly. http://json.org/ There are no limits of the loopy things (the number of consecutive digits in numbers), but I don't consider that a weakness of the standard. Most of the tests that I see do pass completely invalid JSON. http://seriot.ch/json/pruned_results.png

So, 9223372036854775807 is a valid number per the json.org spec, but good luck getting a typical JSON decoder to process that number. A couple I tried returned it as 9.2233720368548e+18, which is not the same number.

Isn't that a limitation of the language and API rather than the parser/decoder? I would guess that most users don't want a JSON decoder that depends on some library for arbitrary-precision numbers and returns such numbers for the user's inconvenience.

The summary table suggests that a real bug was found in about half of the parsers tested, and even a few of those bugs belong to a category that one might choose to ignore: almost any non-trivial function can be made to run out of memory, and lots of functions will crash ungracefully when that happens, rather than correctly free all the memory that was allocated and return an error code, which the caller probably doesn't handle correctly in any case because this case was never tested.

Re: Parsing JSON is a Minefield

#234

Post by zeveb »

haberman wrote: post #164
pjscott wrote: post #117

Earlier quoted context omitted.

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to be straightforward -- no escaping, no backtracking, no ambiguity. I believe the grammar is LL(0), which is a nice touch. And because it's not meant to be human-readable, there's no incentive for people to make their parsers deviate from the strict grammar; e.g…

I'm glad you had a good experience with Protocol Buffers. (I work on the protobuf team at Google). But I would advise a bit of caution. Writing a fully-compliant protobuf implementation is trickier than it looks. I'd recommend running any new parser through the conformance tests; I created these to exercise some of the edge cases. https://github.com/google/protobuf/tree/master/conformance Here are some of the quirks…

> when serializing, all signed integers need to be sign-extended to 64 bits, to support interop between different integer types

FWIW, I think it'd be awesome if the only wire integer format were a bignum, in order to support full interoperability between integer types. Maybe even do that same for floats, too …

Re: Parsing JSON is a Minefield

#235

Post by zeveb »

icebraining wrote: post #100
bjoli wrote: post #59

Earlier quoted context omitted.

JSON can be mapped perfectly to s-expressions. So can xml. Isn't JSON more or less (apart from the commas and colons) just sexprs with a simple schema and different styles of brackets?

I'm not saying it can't be mapped; I'm saying it loses semantics in the translation. For example, how do you represent a boolean in a s-exp, such as that anyone with "the s-exp spec" can unambiguously know that's a boolean?

The problem is that any realistic application imposes more semantics than just the JSON semantics on its data. A particular string isn't just as a string: it's really an address, or a name, or whatever; a particular number isn't just a number, it's an integer, or an ed25519 key, or a percentage: JSON is as incapable of encoding those semantics as are S-expressions (it's also incapable of encoding an ed25519 key as a number anyway — at least portably so!).

Except … canonical S-expressions do have a way to encode type information, if you want to take advantage of it: display hints. [address]"1234 Maple Grove Ln.\nEast Podunk, Idaho" is an atom with a display hint which encodes the type of the string. You could do the same for lists, too: (address "1234 Maple Grove Ln." "East Podunk, Idaho") could be the same.

Does this require application-specific knowledge? Yes. Because it's application-specific. JSON offers you just enough rope to hang yourself: you think, 'oh, these are all the basic types anyone needs' — but people need more than basic types: they need rich types. And the only way to do that is … an application-specific layer which unmarshals data into an application-specific data structure.

It's very appealing to think that an application-agnostic system can usefully process data. That was part of the allure of XML; it's part of the allure of JSON. And while it is true that application-agnostic systems can be a great 85% solution, they tend to break, badly, when they're pushed to the limit. Our time as engineers & scientists would be better spent, IMHO, building systems which enable us to rapidly create application-specific code rather than systems which enable us to slowly create partially-universal code.

Re: Parsing JSON is a Minefield

#236

Post by dagenix »

benzoate wrote: post #200
dagenix wrote: post #190

Earlier quoted context omitted.

So if JSON parsers can't agree on what JSON actually is, that isn't a problem?

I consider it more like behaviour outside of the spec is undefined, and some parsers have bugs.

A good spec should define all those corner cases. With Json, it's possible for two bug free libraries to take the same input and produce a different output.

Re: Parsing JSON is a Minefield

#237

Post by o_____________o »

mchanson wrote: post #128
freshhawk wrote: post #127

It would be great if programmers learned from markdown and json. Here is the lesson: 1. We need something simpler, so I will make a simple solution to this problem 2. Simple should also mean no strict spec, support for versioning or any of those engineer things. All that engineer shit is boring and I can tell myself this laziness is "staying simple" 3. OH SHIT, I was totally right about #1 so this got popular and hav…

Oh yes RFC process always keeps things from having compatibility issues. I definitely never saw any issues with all those XML based standards like SOAP or XSLT.

Wouldn't it be great if we had some ML to transform unnecessary sarcasm?

> RFC processes don't always keep things from having compatibility issues. XML based standards like SOAP or XSLT had many issues.

It would make the internet seem much less passive aggressive. Sarcasm is often just a thin, bitter encoding over simple statements.

Re: Parsing JSON is a Minefield

#238

Post by toast0 »

Someone wrote: post #193
hueving wrote: post #184

Earlier quoted context omitted.

>What would help here is if Alice generated a clean copy of what she thinks she received, and serialized that. I think you just suggested the same thing the OP did.

Rereading it, you may be right. I read “what was parsed” as “as parsed by the json parser”, but chances are the OP meant “by the application layer”. I didnkt read it that way because I don’t see that often in my job. Programs there typically know just enough about the format to do their job, and that job doesn’t include “watch out for external threats” (and they don’t all just use some common library that _does_ know…

Actually, I hadn't meant that, although on reflection, it's the right thing to have meant. This stuff is hard!

Re: Parsing JSON is a Minefield

#239

Post by icebraining »

zeveb wrote: post #235
icebraining wrote: post #100

Earlier quoted context omitted.

I'm not saying it can't be mapped; I'm saying it loses semantics in the translation. For example, how do you represent a boolean in a s-exp, such as that anyone with "the s-exp spec" can unambiguously know that's a boolean?

The problem is that any realistic application imposes more semantics than just the JSON semantics on its data. A particular string isn't just as a string: it's really an address, or a name, or whatever; a particular number isn't just a number, it's an integer, or an ed25519 key, or a percentage: JSON is as incapable of encoding those semantics as are S-expressions (it's also incapable of encoding an ed25519 key as a…

It's very appealing to think that an application-agnostic system can usefully process data. That was part of the allure of XML; it's part of the allure of JSON.

If you don't think an application-agnostic structure is useful, then you wouldn't use S-expressions, you'd use a sequence of arbitrary bytes, and leave all the processing to the application. Using anything else is a compromise between generic and application-specific.

Re: Parsing JSON is a Minefield

#240

Post by fao_ »

nnq wrote: post #226
Annatar wrote: post #220

Earlier quoted context omitted.

Shell and AWK programs have fewest dependencies, are extremely light on resources’ consumption and are extremely fast. When someone’s program emits JSON because the author assumed that programs which ingest that data will also be JavaScript programs, that’s a really bad assumption. It would force the consumer of that data to replicate the environment. This goes against core UNIX principles, as discussed at length in…

> It would force the consumer of that data to replicate the environment. It won't, because JSON is a standard. Imperfect like all standards but practically good enough. And "plain text" just means "an undefined syntax that I have to mostly guess". And nobody "programs" in bash or awk anymore. The "standard scripting languages" for all sane devs are Python or Ruby (and some Perl legacy) and parsing JSON in them is tri…

The standard for plain text output in the terminal is and has always been TSV.
Post reply on HN