Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing malformed JSON

peteris.rocks

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Parsing malformed JSON

#21

Post by drakenot »

bhaak wrote: post #7

Great, after the tag soup of modern browsers are we now also going to see json soup? Sometimes it's obvious what's wrong with malformed data you receive. A classic would be encoding errors. But as soon as you start supporting broken components and APIs, you will never be able to unsupport it. Prime example would be HTML. Granted, in the beginning, it was supposed to be written by humans but that was rather quickly no…

I've written a relatively popular Atom/RSS feed parser for Go [0].

I struggled with this very issue but I ultimately ended up attempting to be robust against out-of-spec feeds. A super strict feed parsing library is less useful than one that can successfully parse certain classes of broken feeds.

It is a fine line to walk -- I won't add a great deal of complexity to support overly broken feeds, but if it is relatively simple to support certain types of common mistakes I'll do it.

[0] https://github.com/mmcdole/gofeed

Re: Parsing malformed JSON

#23

Post by mSparks »

my first reaction would be to parse until you hit a problem. then use a string distance function and a genetic algorithm to find the problematic characters.

in other words. find multiple possibilities that result in valid a json object and choose the one with the shortest distance.

then, of course log out the changes.

I do something similar with csvs. mssql is notorious for spitting out junk inside csv files.

also, i can guess how it was created.

the code is probably in c, and a rare edge case is overwriting memory before it hits the file.

Re: Parsing malformed JSON

#24

Post by hyperman1 »

junke wrote: post #14
k2xl wrote: post #9

I'm hoping nobody actually does this in production. As an academic exercise it is interesting. Maybe I'm old fashioned - I'm all for flexible APIs and all, but to its point. If a customer sends rotten stuff, it should just be rejected with a 40x code. At minimum, check to make sure it is proper JSON... I know that a lot of stream processors will put it into a queue and 200 right away and then process in the backgroun…

I don't deal with such huge files. Honestly, what use case requires 900GB of JSON ?

I got one for you. We have to upload json files containing for a bunch of articles some encoded rules, and the legal text in the law why the encoded rules are what they are.

The law part was supposed to be a few lines of text. Except when they dont know which article to give. In that case they provide the full law text, including scanned pdfs, base64 encoded. All 2GB of it. Basically you have something with the meaning null, encoded in a huge string.

Now the creation of this file was given to a third party, who don't bother with finding out the relevant law, and paste the 2GB blob into every article they modify, just to be sure. At this point we have 500 000 articles in that file. We get a new one every month.

Not fun at all. But it is modern, at least, in the past it was a cobol flat file.

Re: Parsing malformed JSON

#25

Post by treve »

drakenot wrote: post #21
bhaak wrote: post #7

Great, after the tag soup of modern browsers are we now also going to see json soup? Sometimes it's obvious what's wrong with malformed data you receive. A classic would be encoding errors. But as soon as you start supporting broken components and APIs, you will never be able to unsupport it. Prime example would be HTML. Granted, in the beginning, it was supposed to be written by humans but that was rather quickly no…

I've written a relatively popular Atom/RSS feed parser for Go [0]. I struggled with this very issue but I ultimately ended up attempting to be robust against out-of-spec feeds. A super strict feed parsing library is less useful than one that can successfully parse certain classes of broken feeds. It is a fine line to walk -- I won't add a great deal of complexity to support overly broken feeds, but if it is relativel…

I'm doing this with WebDAV too. When I come across a bug that's clearly an implementation problem I weigh how prevalent the software is, how likely they will be able to fix it and if possible I add a user-agent specific workaround so new clients can't rely on the same bug with my server.

Re: Parsing malformed JSON

#27

Post by junke »

hyperman1 wrote: post #24
junke wrote: post #14

Earlier quoted context omitted.

I don't deal with such huge files. Honestly, what use case requires 900GB of JSON ?

I got one for you. We have to upload json files containing for a bunch of articles some encoded rules, and the legal text in the law why the encoded rules are what they are. The law part was supposed to be a few lines of text. Except when they dont know which article to give. In that case they provide the full law text, including scanned pdfs, base64 encoded. All 2GB of it. Basically you have something with the meani…

This looks like TheDailyWTF.com, but thanks.

Re: Parsing malformed JSON

#28

Post by markrages »

drakenot wrote: post #21
bhaak wrote: post #7

Great, after the tag soup of modern browsers are we now also going to see json soup? Sometimes it's obvious what's wrong with malformed data you receive. A classic would be encoding errors. But as soon as you start supporting broken components and APIs, you will never be able to unsupport it. Prime example would be HTML. Granted, in the beginning, it was supposed to be written by humans but that was rather quickly no…

I've written a relatively popular Atom/RSS feed parser for Go [0]. I struggled with this very issue but I ultimately ended up attempting to be robust against out-of-spec feeds. A super strict feed parsing library is less useful than one that can successfully parse certain classes of broken feeds. It is a fine line to walk -- I won't add a great deal of complexity to support overly broken feeds, but if it is relativel…

Nothing new under the sun:

http://www.xml.com/pub/a/2003/01/22/dive-into-xml.html

Post reply on HN