Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

21–30 of 81 posts

Re: Parsing gigabytes of JSON per second

#21

Post by AYBABTME »

cogman10 wrote: post #14
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

It's the same idea as designing hardware for repairability. Sure you can repair something that's welded on, but it's much easier if it's bolted through instead.

Re: Parsing gigabytes of JSON per second

#22

Post by SigmundA »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON.

I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc).

Text files would just be something like { contentType: "text/plain" content : "text here" } while an image could be { contentType: "image/jpeg", content: } you could also add whatever other metadata you want and all of it is retained and easily parseable. Other more structured formats obviously wouldn't just be a blob for content and every system out there would have a structured binary format viewer/editor just like there are text viewers and editors now.

Sqlite is kinda used this way and has some nice properties like indexes and transactions but is relational instead of hierarchal which can be good and bad, not as straight forward to just view or navigate. Protobuf is another used quite a bit now, honestly I don't care just something everyone agree upon that can encode more structure efficiently but still can be easily inspected everywhere.

Doubt this will any time soon but it does seem inevitable in the long run that we figure out a way to send data between system and what a string, number, date etc is and stop having text encoding and escaping issues.

Re: Parsing gigabytes of JSON per second

#23

Post by Zababa »

lumost wrote: post #19
Zababa wrote: post #16

Earlier quoted context omitted.

> If your dataset is large enough to benefit from such a hyperoptimized parser I don't understand that point. Is there an overhead in starting the parsing, which makes regular parsing faster unless you have a large JSON file? If not, why wouldn't you want faster JSON parsing?

Most languages have a built in parser these days, pulling in a c dep can be painful in many build tools and languages. The csimdjson api is sufficiently different to not as idiomatic. Most language json parsers are very fast already. Personally I’m using csimdjson in a project with 100s of TB of json to burn through. This data should not be json formatted, but migrating away from json would require modifications to h…

That's a fair point. Though this could probably be used in interpreters/JIT/C++ projects, which is already a lot. And this gives a good template on how to optimize JSON parsing for other projects.

Re: Parsing gigabytes of JSON per second

#24

Post by throwawaygal7 »

This is pretty great.

Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some.

My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss.

However, we store the data in various relational and document databases. Neither of those use protobuf...

Most importantly, it's business critical to be able to stream this data to data lakes where it can be read by humans. None of the options are going to support protocol buffers, you're either going to have to write a parser (impossible for some) or transform to JSON before ingest (fairly expensive due to some poor choices in how to represent various fields).

It was a sound technical decision to use protoc , and a terrible choice for the business.

I think it would have been much better to use avro, still benefit from schemas but push the work of marshaling JSON back to clients...

Re: Parsing gigabytes of JSON per second

#25

Post by estebarb »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

I worked on a database engine that uses JSON as communication protocol. Sure, most messages are small, but for a system processing JSONs 24/7 it quickly sums a lot.

The main issues see had with our codec was memory allocation and the big number of "if" in the code.

I tried to use simdjson, but there were concerns at that time about portability and long term support. It's nice to see that in retrospective we were wrong.

Re: Parsing gigabytes of JSON per second

#26

Post by kortex »

SigmundA wrote: post #22
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

I don't know why the ASCII characters for record and field separators don't get more love. That is what they are there for. My suspicion is because they aren't type-able, so people rarely encounter them.

Re: Parsing gigabytes of JSON per second

#27

Post by legulere »

SigmundA wrote: post #22
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

You mean like CBOR?

https://en.wikipedia.org/wiki/CBOR

Re: Parsing gigabytes of JSON per second

#28

Post by cogman10 »

AYBABTME wrote: post #21
cogman10 wrote: post #14

Earlier quoted context omitted.

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

It's the same idea as designing hardware for repairability. Sure you can repair something that's welded on, but it's much easier if it's bolted through instead.

It's not the same.

You are almost certainly passing these through tools (even built into the browser) that are doing extra processing to make it more readable.

Let's assume, for example, CBOR ends up taking over JSON. Do you not think browsers wouldn't have a CBOR parser to make it more readable?

This isn't bolt vs welding, this is bolt vs bolt with a washer. Yes there's a small extra step, but not some sort of insurmountable hurdle.

Re: Parsing gigabytes of JSON per second

#29

Post by SigmundA »

legulere wrote: post #27
SigmundA wrote: post #22

Earlier quoted context omitted.

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

You mean like CBOR? https://en.wikipedia.org/wiki/CBOR

Sure, looks good, plenty to choose from, just need to get everyone to agree on one.

Although I do like ones that have some concept of a schema to reduce repeated key size and allow validation.

Re: Parsing gigabytes of JSON per second

#30

Post by orasis »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

protocols have two sides - a producer and a consumer. The consumer often doesn’t have control of the producer, especially in the case of analytics.
Post reply on HN