Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Parsing gigabytes of JSON per second

#41

Post by kozziollek »

secondcoming wrote: post #12
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

You could also potentially use plain CSV or TSV files. Not sexy, but they work.

Oh yes, CSV! Sure! Separated with commas or semicolons? With quotes or without? What decimal separator?

No thanks.

Use Protobufs, Parquet, Avro, etc.

Re: Parsing gigabytes of JSON per second

#42

Post by kabber »

Fast JSON parsers are all fine and good, but take such benchmarks with a grain of salt. JSON, like any format, is useless unless you do something meaningful with it - populate a database, verify the result, or transform it in some useful way for the task at hand. Many of these faster JSON DOM-style parsing libraries store key/value pairs in lists or arrays because it's the most efficient way to do so. But they aren't great at lookup speed. If the JSON parse library is event based, i.e. SAX-style, you have to store it somewhere, and this takes CPU as well.

Re: Parsing gigabytes of JSON per second

#43

Post by yardstick »

secondcoming wrote: post #39
cogman10 wrote: post #14

Earlier quoted context omitted.

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

Indeed, out of all the trillions of HTTP requests that are communicated every day how many get read by a human. Not a lot, yet that's the use-case it's optimised for. Those on the cloud should compute how much HTTP headers count towards their traffic egress bill. Everyone is fine with human-readable as long as it's in English. Binary should be the default. If writing a binary to human decoder is too much for you, you…

HTTP/2 uses a binary encoding instead of text for performance reasons. I prefer the text version myself as it’s easier to write simple tools for (quick clients, telnet based queries, etc).

Re: Parsing gigabytes of JSON per second

#44

Post by Epa095 »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

If the improvements scale, you can still get benefit for smaller use cases. And even if each json is small, if you have millions of them the tiny improvements add up. We parse quite a bit of json, we fetch and store json regularly from external api's. We can't ask for a different format, and for each of the individual requests json makes sense, the response is only a few hundred kbs. But over time there is a lot of data. We convert them to parquet, but each json needs to be read at least once.

Re: Parsing gigabytes of JSON per second

#45

Post by MayeulC »

wodenokoto wrote: post #33
throwawaygal7 wrote: post #24

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement.

Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...

Re: Parsing gigabytes of JSON per second

#46

Post by matja »

wodenokoto wrote: post #33
throwawaygal7 wrote: post #24

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

Tricky part about implementing that is that the protobuf wire-format alone doesn't contain enough information to unambiguously represent the real data types, it also needs the schema (.proto file) to do that.

For example, in the wire-format, a string and a sub-message are encoded as the same type (a blob - varint + sequence of bytes), but using the schema they are clearly interpreted differently.

Sure, it is possible to make a self-describing protobuf message which includes the schema in protobuf representation, but that is a special-case.

Re: Parsing gigabytes of JSON per second

#48

Post by uDontKnowMe »

matja wrote: post #46
wodenokoto wrote: post #33

Earlier quoted context omitted.

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

Tricky part about implementing that is that the protobuf wire-format alone doesn't contain enough information to unambiguously represent the real data types, it also needs the schema (.proto file) to do that. For example, in the wire-format, a string and a sub-message are encoded as the same type (a blob - varint + sequence of bytes), but using the schema they are clearly interpreted differently. Sure, it is possible…

What about Avro then?

Re: Parsing gigabytes of JSON per second

#49

Post by IshKebab »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

Yeah if you have a situation where JSON gets into the megabytes you should definitely stop using JSON! I recommend SQLite for large datasets that you need to put in a file.

I guess if you don't have a choice then it might become a bottleneck. Parsing 300 MB of data can be very slow.

But even with small JSON messages it can become a bottleneck, e.g. check out the Xi editor's issues. They thought it wouldn't be a bottleneck and then found that sound languages don't have insanely optimised JSON parsers like this. Boom. Bottleneck.

Re: Parsing gigabytes of JSON per second

#50

Post by morelisp »

MayeulC wrote: post #45
wodenokoto wrote: post #33

Earlier quoted context omitted.

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...

Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.
Post reply on HN