Parsing JSON is a Minefield • Hacker News Board

Re: Parsing JSON is a Minefield

#101

Post by orf » Sun, Apr 22, 2018, 8:39 PM UTC

ris wrote: post #78

3pt14159 wrote: post #19

Earlier quoted context omitted.

Disagree. I can always make my JSON act like XML if I want to. When I'm following something like JSON API v1.1 I get a lot of the advantages that I'd get from XML with 99% less bloat. You want types? Go for it! There are even official typed JSON options out there. The security / parsing issues with XML alone are enough for me to rule it out. How many critical security issues are the result of libxml? Nokogiri / libxm…

> the whitespace issues of YAML It's not just whitespace issues yaml has - try storing ISO two letter country codes as values and as soon as you get to norway, you've got a boolean. There are many things which are deceptively "simple" where that actually means "I haven't thought about this very much".

Could you not just use 'no'?

Re: Parsing JSON is a Minefield

#102

Post by smel » Sun, Apr 22, 2018, 8:42 PM UTC

It's inevitable, nothing is perfect ... there is only popular things that everyone complains about and things that nobody cares about :D

We're not machines we're more comfortable with messy and forgiving systems.

Do you want to build successful products? be liberal on input and conservative on output. You need to reduce entropy and give people a feeling of magic. It's when you're old and have enough scars on your skin when you learn to hate magic and become control freak :D

Re: Parsing JSON is a Minefield

#103

Post by astrobe_ » Sun, Apr 22, 2018, 8:46 PM UTC

taeric wrote: post #34

bhldr wrote: post #20

Earlier quoted context omitted.

This is also widely considered a bad idea now. Making liberal consumers allows for sloppy producers. Over time this requires new consumers to conform to these sloppy producers to maintain compatibility. Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market.

Do you have a survey or other citation for it being a bad idea? I get that it enables bad behavior, per see. However, the idea of rejecting a customer/client because they did not form their request perfectly seems rather anti customer. Ideally, you'd both accept and correct. But that is the idea, just reworded.

It goes against safety.

"Accept and correct" in the absence of ECC is just delusion if not hubris. The sender could be in a corrupted state and could have sent data it wasn't supposed to send. Or the data could have been corrupted during transfer, accidentally or deliberately. You can't know unless you have a second communication channel (usually an email to the author of the offending piece of software), and what you actually do is literally "guess" the data. How can it go wrong?

Re: Parsing JSON is a Minefield

#104

Post by orf » Sun, Apr 22, 2018, 8:46 PM UTC

gumby wrote: post #44

We had a nasty liberal-in-what-you-accept JSON problem: using JSON to communicate between services written in various languages (Python, Java, Javascript, C++): the python client was simply writing maps out which were automatically serialized into something almost JSON: {'label': 123} (using ' to delimit the label strings, not "). The Javascript JSON parser would silently accept this, as would some of the Java librar…

It was likely that the client was not serialising the data at all, i.e passing a dict to requests.post() in the body parameter. This ends up casting it to a string, hence the single quotes.

Re: Parsing JSON is a Minefield

#105

Post by SOLAR_FIELDS » Sun, Apr 22, 2018, 8:47 PM UTC

helaan wrote: post #40

Someone wrote: post #17

Earlier quoted context omitted.

If your API takes json input, some of those issues are potential security or DoS issues. For example, if you validate your json in your web front-end (EDIT: I used the wrong term. What I meant here is the server-side process that’s in front of your database) and then pass the string received to your json-aware database, you’re likely using two json implementations that may have different ideas about what constitutes…

Reminds me of last years CouchDB bug (CVE-2017-12635) which was caused by two JSON parsers disagreeing on duplicate keys: here it was possible to add a second key with user roles, allowing a user to give admin rights to itself. JSON parser issues are real.

This specific RCE vulnerability was actually given as an explicit example of the consequences of the current state of the specifications.

Re: Parsing JSON is a Minefield

#106

Post by edejong » Sun, Apr 22, 2018, 8:49 PM UTC

jstimpfle wrote: post #32

ProblemFactory wrote: post #28

Earlier quoted context omitted.

XML parsers are necessarily over-complicated for structured data, because it is a text markup language, not a nested data structure language. 123 Hello World Road, 12345 , CA is perfectly sensible XML. The address is not a tree structure or a key-value dictionary - it is free text with optional markup for some words. You can use XML to represent nested data structures with lists and dictionaries, but the parsers and…

Yep, the application to text documents is valid in my eyes, as well. Although there are lighter weight and/or more extensible approaches, like TeX. (update, clarificaton: I mean just the markup syntax, not the compuational model)

From someone who has written TeX macro’s before: you probably mistake the ‘clean’ environment of LaTeX with the core TeX language. The former is reasonable, if very limited, the latter is die-hard “you thought you knew how to program, but this proves you wrong”-material.

XML over TeX any time and LISP-like over XML (with structural macros)

Re: Parsing JSON is a Minefield

#107

Post by drawkbox » Sun, Apr 22, 2018, 8:50 PM UTC

eponeponepon wrote: post #4

This is the one thing that the JSON-against-XML holy warriors need to understand properly. Yes, JSON's less verbose; yes, it's just "plain text" (in as much as there is such a thing); yes, XML makes you put closing tags in - but if you need reliable parsing and rock-solid specifications (and it's reasonably likely that you do, even if you think you don't...), then XML, for all its faults, is very likely the better wa…

Vogons destroyed XML and they would love to destroy JSON. Back away from the JSON vogons, go make another 'simple' format that you put in all your edge cases for complexity. Just try to make a format more simple than JSON, it is based off the basic object, list and basic types string, number, date, bool etc. Where data doesn't fit in those you make it fit or move to another format like YAML, BSON, XML, binary standard like Protobuf or custom binary where needed for say real-time messaging when you control both endpoints always otherwise you have to constantly update a client consumer as well.

JSON is a data and messaging format meant to simplify. If you can't serialize/deserialize to/from JSON then your format might be too complex, and if it doesn't exactly fit in JSON just put the value in a key and add a 'type' or 'meta' key that allows you to translate to and from it. If binary store it in base64, if it is a massive number put it in a string and a type next to it to convert to and from. JSON is merely the messenger, don't shoot it. JSON is so simple it can roll from front-end to back-end where parsing XML/binary in some areas is more of a pain especially for third party consumers.

JSON being simple actually simplifies systems built with it which is a good thing for engineers that like to take complexity and make it simple rather than simplicity to complexity like a vogon.

Re: Parsing JSON is a Minefield

#108

Post by ChrisSD » Sun, Apr 22, 2018, 8:54 PM UTC

toast0 wrote: post #96

izacus wrote: post #79

Earlier quoted context omitted.

How do Protocol Buffers (which I see used quite alot in similar environments as JSON) compare? Anyone has experience in the format?

My experience with json and similar formats is that most of the complexity arrises from using delimited strings instead of length prefixed strings, and the exciting escaping that results. If the strings are character strings instead of byte strings, you get to add an extra layer of character encoding excitement. PHP serialization is better here, everything is type:value or type:length:value, although strings do have…

The problem is that formats like JSON are designed to be human readable and writable. Length prefixing is a non starter here.

Protobuf and similar are binary formats so don't have this limitation.

Re: Parsing JSON is a Minefield

#109

Post by edejong » Sun, Apr 22, 2018, 8:54 PM UTC

If you think parsing JSON is hard, try parsing/generating streaming JSON while limiting memory bandwidth requirements. Fun exercise, with a push-down automaton.

Re: Parsing JSON is a Minefield

#110

Post by IshKebab » Sun, Apr 22, 2018, 9:02 PM UTC

SamReidHughes wrote: post #91

skywhopper wrote: post #52

While this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serializa…

No, this is not true of many reasonable formats. You don't have to make an obtusely nontrivial format to encode the data JSON does.

Most of the things they test are true of any text-based format, and many of them are true of any serialisation format. E.g. 100000 opening brackets. You could do the same in XML for example and I expect many parsers would fail.