Parsing JSON is a Minefield • Hacker News Board

Re: Parsing JSON is a Minefield

#251

Post by zeveb » Tue, Apr 24, 2018, 3:39 PM UTC

ChrisSD wrote: post #108

toast0 wrote: post #96

Earlier quoted context omitted.

My experience with json and similar formats is that most of the complexity arrises from using delimited strings instead of length prefixed strings, and the exciting escaping that results. If the strings are character strings instead of byte strings, you get to add an extra layer of character encoding excitement. PHP serialization is better here, everything is type:value or type:length:value, although strings do have…

The problem is that formats like JSON are designed to be human readable and writable. Length prefixing is a non starter here. Protobuf and similar are binary formats so don't have this limitation.

> The problem is that formats like JSON are designed to be human readable and writable. Length prefixing is a non starter here.

Canonical S-expression are both human-readable & length-prefixed. They do this by have an advanced representation which is human-friendly:

    (data (looks "like this" |YWluJ3QgaXQgY29vbD8=|))

And a canonical representation which is length-prefixed:

    (4:data(5:looks9:like this14:ain't it cool?))

Re: Parsing JSON is a Minefield

#252

Post by nnq » Wed, Apr 25, 2018, 8:03 AM UTC

fao_ wrote: post #240

nnq wrote: post #226

Earlier quoted context omitted.

> It would force the consumer of that data to replicate the environment. It won't, because JSON is a standard. Imperfect like all standards but practically good enough. And "plain text" just means "an undefined syntax that I have to mostly guess". And nobody "programs" in bash or awk anymore. The "standard scripting languages" for all sane devs are Python or Ruby (and some Perl legacy) and parsing JSON in them is tri…

The standard for plain text output in the terminal is and has always been TSV.

...and how does one do nested/tree-like data in TSV? One doesn't, only just ad-hoc invents something akin to sexps or json but unstandardized. Real-world data rarely has a natural fit for "tabl

Re: Parsing JSON is a Minefield

#253

Post by zmix » Wed, Apr 25, 2018, 10:54 PM UTC

eadmund wrote: post #134

mchanson wrote: post #128

Earlier quoted context omitted.

Oh yes RFC process always keeps things from having compatibility issues. I definitely never saw any issues with all those XML based standards like SOAP or XSLT.

A lot of that is because XML is objectively insane: it's a monumentally over-specified version of something that a sane community would have sketched out on the back of a cocktail napkin. XML is S-expressions done wrong. It's a massive amount of ceremony & boilerplate, IMHO due to the pain of dealing with dynamic data in static languages. It's basically the Java of data-transfer languages. And it shouldn't even be us…

> And it shouldn't even be used for data transfer: it's a markup language

XML could never be used for data transfer. That is being done by the protocol. That would be, in most of the XML cases: HTTP(S).

GET http://en.wikipedia.org/wiki/Rolling_Stones/wp:discography/w... GET http://store.steamcommunity.com/profile/myprofile/games.xml/...

Wow! That's a beauty! And that's only because XML is BOTH a document and a data structure. It has two personalities, but only one identity. And it is not schizophrenic about it. It's always clear.

> A lot of that is because XML is objectively insane

I don't find "everything is a node" to be insane. It's like "Everything is a file" followed through up to the atomic value.

/net/host/volume/directory/file.xml/document-node/some/other/node/attribute

or

/net/host/volume/directory/file.xml//all-nodes[@where-this-attributes-value="foobar"]

Looks like a perfect match for both command line as well as RESTful access.

> monumentally over-specified

The XML spec, while having a healthy size, is not overly big: https://www.w3.org/TR/xml/

Do not confuse the additional specs like XSL, XPath and XQuery as the "XML" spec. These are your toolbox. And their volume is in no way bigger than any of the frameworks, programmers use. Also XSD is not really part of the XML spec. You don't need it in many cases.

It's a meta language, that consists of a simple convention: Elements and Attributes. You name them what you want and get a document, that, at the same time is a queriable datastructure. But I've said that already...

Re: Parsing JSON is a Minefield

#254

Post by [deleted] » Fri, Apr 27, 2018, 3:14 PM UTC

keymone wrote: post #39

i wish there was a chance for EDN[1] to replace JSON. it's a shame the industry defaulted to a subset of javascript as a data notation format considering all it's shortcomings =/ yeah, i get it, "but it has native support in all browsers" is a valid argument, i just wish it wasn't. [1] https://github.com/edn-format/edn

[deleted]

Re: Parsing JSON is a Minefield

#255

Post by draegtun » Fri, Apr 27, 2018, 3:16 PM UTC

keymone wrote: post #39

i wish there was a chance for EDN[1] to replace JSON. it's a shame the industry defaulted to a subset of javascript as a data notation format considering all it's shortcomings =/ yeah, i get it, "but it has native support in all browsers" is a valid argument, i just wish it wasn't. [1] https://github.com/edn-format/edn

It's even a bigger shame that Crockford wasn't able to go with Rebol (instead of "discovering" JSON), which he was originally pushing/planning :(

Re: Parsing JSON is a Minefield

#256

Post by meuk » Sat, Apr 28, 2018, 12:05 PM UTC

jacobparker wrote: post #225

arghwhat wrote: post #213

Earlier quoted context omitted.

JSON parsers all agree what JSON actually is (except for the brown colored fields in the compat matrix, which are actual JSON compat bugs). JSON parsers may also permit various additional variations which are very explicitly not JSON. This means that they may accept a handcrafted, technically invalid JSON document. However, a JSON encoder may never generate a document containing such "extensions", as this would not b…

> As the robustness principle goes [...] "The Harmful Consequences of the Robustness Principle" https://tools.ietf.org/html/draft-thomson-postel-was-wrong-0...

I want to upvote this so many more times than I'm able to.

The principle is even more harmful, because it sounds so logical. If many JSON parser accept your JSON object which is not valid JSON, any new parser that doesn't accept it will be booed as a faulty parser.

If your input is not according to spec, throw an error. The sender is wrong. They deserve to know it and need to fix their output.

Re: Parsing JSON is a Minefield

#257

Post by fao_ » Wed, May 02, 2018, 5:14 PM UTC

nnq wrote: post #252

fao_ wrote: post #240

Earlier quoted context omitted.

The standard for plain text output in the terminal is and has always been TSV.

...and how does one do nested/tree-like data in TSV? One doesn't, only just ad-hoc invents something akin to sexps or json but unstandardized. Real-world data rarely has a natural fit for "tabl

How many standard UNIX programs do you know that output tree data?

And almost always you can find an alternative to non-tree-based representation without having to reinvent sexprs. Look at how ps represents the process tree, for example.