Live data from Hacker News

Parsing JSON is a Minefield

seriot.ch

51–60 of 257 posts

Re: Parsing JSON is a Minefield

#51

Earlier quoted context omitted.

XML parsers are necessarily over-complicated for structured data, because it is a text markup language, not a nested data structure language. 123 Hello World Road, 12345 , CA is perfectly sensible XML. The address is not a tree structure or a key-value dictionary - it is free text with optional markup for some words. You can use XML to represent nested data structures with lists and dictionaries, but the parsers and…

Yep, the application to text documents is valid in my eyes, as well. Although there are lighter weight and/or more extensible approaches, like TeX. (update, clarificaton: I mean just the markup syntax, not the compuational model)

> TeX

Dear god no. I use and love (La)TeX daily to write documents. But as a markup format for data that's supposed to be processed in any way, other than being fed to a TeX engine, it's absolutely terrible. You can't even really parse a TeX document; with all the macros it really is more a program than a document. XML is far from perfect, but it works well as a markup and data exchange format that is well-specified.

Re: Parsing JSON is a Minefield

#52
While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that:

1) implementation matters

2) "simple" specs never really are

It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serialization format, encoding standard, or IPC protocol and it would remain true:

" is not the easy, idealised format as many do believe. [There are not] two libraries that exhibit the very same behaviour. Moreover, [...] edge cases and maliciously crafted payloads can cause bugs, crashes and denial of services, mainly because libraries rely on specifications that have evolved over time and that left many details loosely specified or not specified at all."

Re: Parsing JSON is a Minefield

#53
post #21

I think the idea of humans sharing a language with computers is problematic at a fundamental level. (the whole $dataformat "easy to read for humans") It becomes a source of never ending lose-lose compromises where the more points you give to human convenience the more points you take away from machine convenience and vice versa. Then you end up having to "settle" for something in between that is just ambiguous and pr…

>> I don't hear people wanting a human readable text representation of their audio, video or images. This is, in fact, a huge concern for people who think about accessibility.

Ok but what I'm talking about is a little more specific.

Talking about "human readability" of JSON and XML is a little bit like talking about human readability of JPEG or MP3.

Chasing after it creates a lot of problems.

Formats like JSON and XML often carry lots of textual information so it's tempting to want them to be like "just like text but with some extra stuff" but that creates its own problems.

So it would be interesting to have something like JSON but philosophically treat it like MP3. Meaning, don't assume that humans must fiddle with the bytes in a text editor with great ease so that the representation can be designed without the influence of "would the raw bytes look pretty to people?".

Re: Parsing JSON is a Minefield

#55

This is the one thing that the JSON-against-XML holy warriors need to understand properly. Yes, JSON's less verbose; yes, it's just "plain text" (in as much as there is such a thing); yes, XML makes you put closing tags in - but if you need reliable parsing and rock-solid specifications (and it's reasonably likely that you do, even if you think you don't...), then XML, for all its faults, is very likely the better wa…

If you think XML doesn't suffer from all the same issues, you haven't used it enough. I'd use protobuf for something that needs stict serialization and parsing.

[deleted]

Re: Parsing JSON is a Minefield

#56
Before JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. Simply exchanging data was a project in itself.

At least JSON and XML are text based when it comes to data exchange. Back in the day before APIs that needed to exchange data cleanly, without JSON/XML, exchanging data was not only a minefield but one with constant carpet bombing. The fact that edge cases that are rarely run into is all that is left of data exchange issues is a huge advancement.

What is great is in most cases JSON works fantastic and simplifies data exchange and APIs all the way to front ends and backends. XML is available if needed. So are binary standard formats now for really compact areas like messaging for performance that humans may never see or you may never have a third party that needs to parse it. The task of parsing xml in client side javascript is not fun especially, neither is binary parsing where adding a value can break the whole object, JSON keys can come and go.

The engineer can choose the tool for the job but there better be a good reason to use anything over simple JSON, almost any problem can be solved with it. Engineers should aim to take data complexity and make it as simple as possible, not take simple and make it complex for job security, real engineers always move more simple when possible and away from vogon ways.

For data that is exchanged between services and front-end/backend, JSON is the simplified format that makes things move faster. XML got tarred and devolved into vogon sludge with SOAP services and nested namespacing/schemas but still is needed in some areas. Binary standard formats when you control both sides or noone else needs to connect to it or you don't need it on the front end maybe or possibly you need performant real-time messaging. There is also YAML if you need more typing or BSON where binary needed but still simple. All formats have good uses and bad but using binary when JSON will suffice is not being as simple as possible.

JSON is easy to get around and is more lightweight, if you run into a problem you can just restructure your JSON to make it work where binary or XML take more work to change without breaking changes, especially downstream causing many more versions and conversions. JSON is a data messaging format meant to simplify. Most of the issues in the OP article could be solved storing the values in a string with a "type" or "info" key that allows conversion in the backend i.e. long numbers or hex etc or storing binary as base64 etc.

JSON is based on basic CS types in objects, lists, simple data types like string, number, bool, date, this makes for a simplifying of all systems that serialize and deserialize to it. JSON helps spread simplicity while being dynamic.

JSON works best with ever changing dynamic data/code/projects we build today and in seconds you can be consuming data from third party APIs faster than any other format and more simplistically, that is why it won.

Re: Parsing JSON is a Minefield

#57
post #17

This is interesting and important in one way: anything poorly specified will eventually cause a problem for someone, somewhere. That being said, my first response was to complete the title, ". . . yet it remains useful and nearly trouble-free in practice." There's a lot of, "You know what I mean!" in the JSON definition, but in most cases, we really do know what Crockford means.

If your API takes json input, some of those issues are potential security or DoS issues. For example, if you validate your json in your web front-end (EDIT: I used the wrong term. What I meant here is the server-side process that’s in front of your database) and then pass the string received to your json-aware database, you’re likely using two json implementations that may have different ideas about what constitutes…

I sure hope you don’t just put random user provided blobs in your database, even if they’re validated. Also, how do you validate without parsing? If it’s parsed, might as well serialize again when saving to the DB.

Re: Parsing JSON is a Minefield

#58
post #17

This is interesting and important in one way: anything poorly specified will eventually cause a problem for someone, somewhere. That being said, my first response was to complete the title, ". . . yet it remains useful and nearly trouble-free in practice." There's a lot of, "You know what I mean!" in the JSON definition, but in most cases, we really do know what Crockford means.

If your API takes json input, some of those issues are potential security or DoS issues. For example, if you validate your json in your web front-end (EDIT: I used the wrong term. What I meant here is the server-side process that’s in front of your database) and then pass the string received to your json-aware database, you’re likely using two json implementations that may have different ideas about what constitutes…

I think it is rather common sense to do data validation on backend instead of frontend. What matters is that backend always acts as the source of truth, it doesn't really matter if frontend and backend are inconsistent as long as we know that backend data is correct.

Re: Parsing JSON is a Minefield

#59

Earlier quoted context omitted.

If I needed such strictness in parsing I'd fall back to s-expressions, not something that requires a parser like this: $ ls -lah /usr/lib/libxml2.so.2.9.8 -rwxr-xr-x 1 root root 1,4M mar 27 17:46 /usr/lib/libxml2.so.2.9.8

But that's the point, isn't it? S-expressions are light because they define very little, it's only a tree of undefined blobs of data (atoms). It's even more limited than JSON.

JSON can be mapped perfectly to s-expressions. So can xml.

Isn't JSON more or less (apart from the commas and colons) just sexprs with a simple schema and different styles of brackets?

Post reply on HN