Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

31–40 of 81 posts

Re: Parsing gigabytes of JSON per second

#31

Post by SigmundA »

kortex wrote: post #26
SigmundA wrote: post #22

Earlier quoted context omitted.

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

I don't know why the ASCII characters for record and field separators don't get more love. That is what they are there for. My suspicion is because they aren't type-able, so people rarely encounter them.

Still have the issue of binary data encoding as well since the data will have those bytes in it so they need to be escaped vs a format that defines how to encoding binary as is efficiently (length delimited).

Re: Parsing gigabytes of JSON per second

#32

Post by strictfp »

legulere wrote: post #27
SigmundA wrote: post #22

Earlier quoted context omitted.

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

You mean like CBOR? https://en.wikipedia.org/wiki/CBOR

There's also Minecraft NBT. It's a bit obscure but quite nice. https://minecraft.fandom.com/wiki/NBT_format

Re: Parsing gigabytes of JSON per second

#33

Post by wodenokoto »

throwawaygal7 wrote: post #24

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader.

I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

Re: Parsing gigabytes of JSON per second

#34

Post by pablodavila »

SigmundA wrote: post #22
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

Something like UBF might be interesting to you: https://ubf.github.io/ubf/

Re: Parsing gigabytes of JSON per second

#35

Post by llamajams »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

I'm working on an embedded vision/navigation system. The core app itself generates about 50mb a minute of metadata alone in the simplest scenario. It's supposed to be long running system. There is one case that generates just over a TB.

Now you may say we should optimize this or that but I'm not an architect and I have no say, and it's just what I gotta deal with:(

Re: Parsing gigabytes of JSON per second

#36

Post by pickledish »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I guess an example would be along the lines of protobufs, e.g. Cap’n Proto

https://capnproto.org/

(which I have never used but really enjoy just for its charming website design alone)

Re: Parsing gigabytes of JSON per second

#38

Post by gravypod »

throwawaygal7 wrote: post #24

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…

I've written translation layers for such systems and it's not too bad. See this project from $job - 1: https://github.com/CaperAi/pronto

It allowed us to have a single model for storage in the DB, for sending between services, and syncing to edge devices.

Re: Parsing gigabytes of JSON per second

#39

Post by secondcoming »

cogman10 wrote: post #14
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

Indeed, out of all the trillions of HTTP requests that are communicated every day how many get read by a human. Not a lot, yet that's the use-case it's optimised for.

Those on the cloud should compute how much HTTP headers count towards their traffic egress bill.

Everyone is fine with human-readable as long as it's in English.

Binary should be the default. If writing a binary to human decoder is too much for you, you're in the wrong job.

Re: Parsing gigabytes of JSON per second

#40

Post by the8472 »

EwanToo wrote: post #5
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON

Converting a text format to a compact in-memory data structure takes extra CPU cycles. (de)compression takes extra cycles. The point of using a binary format is to achieve the same result while avoiding that overhead. For some data formats compression also has the downside that it prevents seeking.
Post reply on HN