Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

11–20 of 81 posts

Re: Parsing gigabytes of JSON per second

#11

Post by jamal-kumar »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

I've dealt with hundreds of gigabytes of JSON data like that in the fintech world

And yeah I just streamed the data thru the processor and avoid malloc at all costs and it works in seconds

I bet this method works even faster though somehow if they felt compelled to throw research money at all at it

Re: Parsing gigabytes of JSON per second

#12

Post by secondcoming »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

You could also potentially use plain CSV or TSV files. Not sexy, but they work.

Re: Parsing gigabytes of JSON per second

#13

Post by m_mueller »

dan-robertson wrote: post #8
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Well json is everywhere so I think it’s mostly too late for companies to change all their internal protocols and apis to not use it. And note that a big advantage of json is that many applications can process it without a schema—you don’t need to know that the username field is a certain length, type and at a particular place. It is also easier to know if json is valid than a random binary format. The advantage of so…

yep. once you’ve had to deal with XMLs that come in without a schema alongside, you start appreciating getting a JSON. at least that properly distinguishes strings, nulls, integers and decimals.

Re: Parsing gigabytes of JSON per second

#14

Post by cogman10 »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction.

It seems like a notion that started in the early internet and just refuses to die.

Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http2+ being "it's a binary format! On NO!".

JSON seems like a similar relic. We use it not because it's fast, but because we like the idea that it's easy to decode (Even if in practice that almost never happens).

Re: Parsing gigabytes of JSON per second

#15

Post by Zababa »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

If you're at a very large scale, maybe the reduced server cost by parsing JSON this way can "finance" an engineer? If you're starting to reach the limits of JSON, maybe it could also help you avoid a transition for a while longer? At least those are the obvious cases for me.

If it's a drop-in replacement for other JSON parsers, I think this could have a huge impact. There is a lot of value to gain by optimizing the fundamentals

Re: Parsing gigabytes of JSON per second

#16

Post by Zababa »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

> If your dataset is large enough to benefit from such a hyperoptimized parser

I don't understand that point. Is there an overhead in starting the parsing, which makes regular parsing faster unless you have a large JSON file? If not, why wouldn't you want faster JSON parsing?

Re: Parsing gigabytes of JSON per second

#17

Post by suction »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

I worked for small(ish) music supplier of Spotify, not a major label like Universal etc. at all - they were receiving a daily 3-4 Gb JSON file (uncompressed) of yesterday's activity. Things get big, fast, nowadays.

Re: Parsing gigabytes of JSON per second

#18

Post by NicoJuicy »

cogman10 wrote: post #14
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

I regulary get requests from QA why something is not working. Just watching the dumped request and seeing what is wrong ( eg. Configuration) is a bless.

Re: Parsing gigabytes of JSON per second

#19

Post by lumost »

Zababa wrote: post #16
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

> If your dataset is large enough to benefit from such a hyperoptimized parser I don't understand that point. Is there an overhead in starting the parsing, which makes regular parsing faster unless you have a large JSON file? If not, why wouldn't you want faster JSON parsing?

Most languages have a built in parser these days, pulling in a c dep can be painful in many build tools and languages. The csimdjson api is sufficiently different to not as idiomatic. Most language json parsers are very fast already.

Personally I’m using csimdjson in a project with 100s of TB of json to burn through. This data should not be json formatted, but migrating away from json would require modifications to hundreds of different systems.

Re: Parsing gigabytes of JSON per second

#20

Post by djbebs »

cogman10 wrote: post #14
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

Makes it easier to troubleshoot.
Post reply on HN