Parsing malformed JSON

#1

Post by p8donald » Thu, Nov 17, 2016, 2:32 PM UTC

Re: Parsing malformed JSON

#2

Post by PaulHoule » Thu, Nov 17, 2016, 2:54 PM UTC

Malformed data is a scalability problem. Unusual failure modes from coding problems to random bit flips become inevitable as the data volume approaches infinity.

Re: Parsing malformed JSON

#3

Post by junke » Thu, Nov 17, 2016, 2:56 PM UTC

> I had this "JSON" file sent to me

Why? By whom? Did you complain loudly?

Re: Parsing malformed JSON

#4

Post by thwd » Thu, Nov 17, 2016, 3:00 PM UTC

PaulHoule wrote: post #2

Malformed data is a scalability problem. Unusual failure modes from coding problems to random bit flips become inevitable as the data volume approaches infinity.

Agreed, but the payload from the article doesn't seem to have suffered from astral radiation. Rather, random attempts at quote-escaping by someone who doesn't understand what they're doing. Also notice the "nan" value -- JSON has no concept of NaN.

Re: Parsing malformed JSON

#5

Post by wccrawford » Thu, Nov 17, 2016, 3:13 PM UTC

It's a neat trick, but not something I'd deploy into production. If I have to try to guess at what the customer is sending me, I'm not going to apply it to their account.

In an emergency, I might hand-edit it and make it right, but I'd absolutely insist that further files be in the correct format.

Re: Parsing malformed JSON

#6

Post by k__ » Thu, Nov 17, 2016, 3:16 PM UTC

wccrawford wrote: post #5

It's a neat trick, but not something I'd deploy into production. If I have to try to guess at what the customer is sending me, I'm not going to apply it to their account. In an emergency, I might hand-edit it and make it right, but I'd absolutely insist that further files be in the correct format.

Isn't this used mainly in editors that want to provide some hints even for JSON you didn't finished yet.

Re: Parsing malformed JSON

#7

Post by bhaak » Thu, Nov 17, 2016, 3:18 PM UTC

Great, after the tag soup of modern browsers are we now also going to see json soup?

Sometimes it's obvious what's wrong with malformed data you receive. A classic would be encoding errors.

But as soon as you start supporting broken components and APIs, you will never be able to unsupport it.

Prime example would be HTML. Granted, in the beginning, it was supposed to be written by humans but that was rather quickly not a major obstacle anymore and even a human can produce valid HTML with the help of a syntax checker.

Re: Parsing malformed JSON

#8

Post by anentropic » Thu, Nov 17, 2016, 3:19 PM UTC

just send the file back where it came from

Re: Parsing malformed JSON

#9

Post by k2xl » Thu, Nov 17, 2016, 3:24 PM UTC

I'm hoping nobody actually does this in production. As an academic exercise it is interesting.

Maybe I'm old fashioned - I'm all for flexible APIs and all, but to its point. If a customer sends rotten stuff, it should just be rejected with a 40x code.

At minimum, check to make sure it is proper JSON... I know that a lot of stream processors will put it into a queue and 200 right away and then process in the background, but I don't think that ensuring it is at least JSON and doesn't have a content size of more than X could be too intensive.

In this case, if the data was already accepted and you've got no choice but to deal with it, you've gotta do what you got to do. I've been there, and it ain't fun cleaning up a 900 GB JSON file.

Re: Parsing malformed JSON

#10

Post by colanderman » Thu, Nov 17, 2016, 3:24 PM UTC

PaulHoule wrote: post #2

Malformed data is a scalability problem. Unusual failure modes from coding problems to random bit flips become inevitable as the data volume approaches infinity.

Yes, and that's a problem to be solved at the transport and storage layers, not the application layer.