Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

Post reply on HN
Print view View original

201–210 of 257 posts

Re: Parsing JSON is a Minefield

#201

Post by peoplewindow »

ajross wrote: post #38
borplk wrote: post #26

Earlier quoted context omitted.

You have misunderstood the point I was trying to make. I'm saying the idea that we as humans should view/consume the exact same thing raw and byte-by-byte without any transformations in between as computers do creates a problematic trade-off because that representation now has two very different masters, humans and machines. If you want to make that representation pretty and friendly for humans it becomes hard to par…

No, I'm pretty sure I took that point. It's just that... that isn't the problem. All formats get messy for the same reason that all software designs get messy. It's just that messy software is amenable to replacement, while messy formats leave their garbage in public to scream about on HN. And like I said, we've been where you want to be: .xls, in particular, is actually a very simple format at its core and easily in…

You're using a file format designed three decades ago to make arguments about what we should be doing today.

XLS as a format sucks because:

a) It was undocumented for most of its life

b) Undocumented even inside Microsoft because it consisted partly of memory dumps from the app

c) Was heavily optimised for fast loading and saving on very slow machines

100% of uses of JSON, translated to binary, would not suffer those issues. They'd have documented schemas, at least internally, they wouldn't be created by memcpy to disk, and they wouldn't be stuffed with app-specific loading optimisations.

Re: Parsing JSON is a Minefield

#203

Post by freshhawk »

freshhawk wrote: post #127

It would be great if programmers learned from markdown and json. Here is the lesson: 1. We need something simpler, so I will make a simple solution to this problem 2. Simple should also mean no strict spec, support for versioning or any of those engineer things. All that engineer shit is boring and I can tell myself this laziness is "staying simple" 3. OH SHIT, I was totally right about #1 so this got popular and hav…

Ugh, it is too late to do anything about that formatting. Sorry.

Re: Parsing JSON is a Minefield

#204

Post by iainmerrick »

pjscott wrote: post #129
iainmerrick wrote: post #126

Earlier quoted context omitted.

There is no such quirk. Trailing commas are simply not standard JSON (sadly). Even if they were, it would still be easy to parse!

You're right in the same technically-correct sense that parsing HTML 4 is easy: it's just SGML! And as long as you don't have to support any of the crazy deviations from the spec that some people have come to rely on, that's fine. There's a whole spectrum of unofficial parser "helpfulness" here, with HTML 4 being an extreme case of parsers filled with hacks to deal with existing broken data, protobufs being an extrem…

Yeah, that spectrum is a good way to think about it.

JSON hits a sweet spot of being very easy for computers to deal with almost all the time, while also being reasonably easy for humans to read and write.

I was going to add “if you started with that as the spec, it wouldn’t be hard to design something better than JSON” but real examples like YAML are pretty awkward, so probably it’s a harder problem than it seems.

Re: Parsing JSON is a Minefield

#205

Post by jstimpfle »

dagenix wrote: post #192
jstimpfle wrote: post #7

Earlier quoted context omitted.

Xml sort-of died (in many domains) because of its insane complexity and its redundant ways of specifying relations (child relationships vs explicit relationships, tag name data, attributes data, body data), lack of legibility, parser performance (probably an inherent problem due to hierarchical representation?) and other issues like even meaning of whitespace. >50% of the JSON or Xml I've seen would actually be much…

"Insane complexity"? You mean that it actually has a spec instead of the back of a business card with no implementations that agree on what is actually valid?

Yes. No. What it does is too complicated. And I hear that consistent implementations are not a reality for Xml, either (at least regarding implemented features).

Re: Parsing JSON is a Minefield

#206

Post by Mikhail_Edoshin »

iainmerrick wrote: post #126
pjscott wrote: post #117

Earlier quoted context omitted.

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to be straightforward -- no escaping, no backtracking, no ambiguity. I believe the grammar is LL(0), which is a nice touch. And because it's not meant to be human-readable, there's no incentive for people to make their parsers deviate from the strict grammar; e.g…

There is no such quirk. Trailing commas are simply not standard JSON (sadly). Even if they were, it would still be easy to parse!

It would be much easier to emit if they were standard. For example, in XML each element is self-contained; I can pour it into the data stream without knowing if it is preceded or followed by a sibling. With JSON I have to manage the context.

Re: Parsing JSON is a Minefield

#207

Post by Mikhail_Edoshin »

iainmerrick wrote: post #204
pjscott wrote: post #129

Earlier quoted context omitted.

You're right in the same technically-correct sense that parsing HTML 4 is easy: it's just SGML! And as long as you don't have to support any of the crazy deviations from the spec that some people have come to rely on, that's fine. There's a whole spectrum of unofficial parser "helpfulness" here, with HTML 4 being an extreme case of parsers filled with hacks to deal with existing broken data, protobufs being an extrem…

Yeah, that spectrum is a good way to think about it. JSON hits a sweet spot of being very easy for computers to deal with almost all the time, while also being reasonably easy for humans to read and write. I was going to add “if you started with that as the spec, it wouldn’t be hard to design something better than JSON” but real examples like YAML are pretty awkward, so probably it’s a harder problem than it seems.

I think it's the other way round: JSON mimics the syntax of JavaScript literals and the syntax was meant to be easy for humans to write and read. Not super-easy, because it has to be easily parseable as well, but still the original use case of this notation is to write relatively short pieces of code. This is why it seems easy to people and they assume it's easy for computers as well, while in fact JSON is hard for computers to emit because of the trailing comma issue (not super-hard, but harder than XML, which has no separators: elements are self-contained).

Re: Parsing JSON is a Minefield

#208

Post by Mikhail_Edoshin »

krapp wrote: post #132
guntars wrote: post #57

Earlier quoted context omitted.

I sure hope you don’t just put random user provided blobs in your database, even if they’re validated. Also, how do you validate without parsing? If it’s parsed, might as well serialize again when saving to the DB.

If it's parsed, why even store in in a database as JSON at all? If you don't do that... then multiple possible JSON parsers aren't a problem.

Use case: I sync local data with web API. I do not use all the data I receive, only a few bits, but if I modify them, I have to send a complete object back to the server with all the other data. The simplest way to do this is to store the original JSON.

The CardDAV and CalDAV are not JSON, but their specification also requires you to preserve the whole vCard if you ever want to send your changes back to the server. CardDAV data may be accessed by multiple apps and they are allowed to add their private properties; any app that deals with vCards must preserve all properties, including those it doesn't understand or use.

Re: Parsing JSON is a Minefield

#209

Post by Mikhail_Edoshin »

eadmund wrote: post #134
mchanson wrote: post #128

Earlier quoted context omitted.

Oh yes RFC process always keeps things from having compatibility issues. I definitely never saw any issues with all those XML based standards like SOAP or XSLT.

A lot of that is because XML is objectively insane: it's a monumentally over-specified version of something that a sane community would have sketched out on the back of a cocktail napkin. XML is S-expressions done wrong. It's a massive amount of ceremony & boilerplate, IMHO due to the pain of dealing with dynamic data in static languages. It's basically the Java of data-transfer languages. And it shouldn't even be us…

Please. XML specification is much shorter than that of YAML, for example, even though XML 1.0 includes a simple grammar-based validation spec (DTD). "A markup language"? What does it mean? Are there any special "data-transfer languages" we neglect? :) Data gets serialized; we need to mark different parts of it; XML can totally do it. For some cases it's not the best fit, but nothing is.

Re: Parsing JSON is a Minefield

#210

Post by tonyg »

djur wrote: post #199
eadmund wrote: post #180

Earlier quoted context omitted.

> Is SGML a symbol, identifier, a quoteless string? It's a sequence of bytes — a string, if you like. > How do I know when parsing the 'entry' field that what follows is going to be a list of key/value pairs without parsing the whole expression? You wouldn't, and as a parser you wouldn't need to. The thing which accepts the parsed lists of byte-sequences would need to know what to do with whatever it's given, but tha…

I hope you understand that those questions were rhetorical -- they're questions that do not need to be asked about the equivalent JSON representation. Questions developers don't have to ask each other about the data they're sending each other. The canonical S-expression representation solves some of the problems JSON has, true, but the example you provided is not a canonical S-expression. It wouldn't make sense for i…

Interpreted as a Rivest S-expression, the example given above conforms to the "advanced transport representation" [1], and so can automatically and straightforwardly be converted to the "canonical representation" [2].

In an important sense, then, I'd claim that it is a "canonical S-expression".

The reason this works is because SPKI S-expressions aren't just a grammar for a syntax, they also come with [3] a total /equivalence relation/, which is exactly what JSON lacks and which is what makes JSON such a pain to work with.

In other words, SPKI S-expressions have a semantics. JSON doesn't.

Lots of other "modern" data languages also lack equivalence relations, making them similarly difficult to use at scale.

[ETA: Of course, your point about lacking common data types is a good one! My fantasy-land ideal data language would be something drawing from both SPKI S-expressions and BitTorrent's "bencoding", which includes integers and hashes as well as binary blobs and lists.]

---

[1] Section 6.3 of http://people.csail.mit.edu/rivest/Sexp.txt

[2] Section 6.1 of http://people.csail.mit.edu/rivest/Sexp.txt

[3] The SPKI S-expression definition is still a draft and suffers a few obvious problems - ASCII-centrism and the notion of a "default MIME type" being two major deficits. Still, I'd love to see the document revived, updated, and completed. Simply having an equivalence relation already lifts it head and shoulders above many competing data languages.

Post reply on HN