Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Parsing gigabytes of JSON per second

#61

Post by koolba »

ec109685 wrote: post #60
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Why do you think human readable adds appreciable overhead? If you want to create a flexible interchange format, that is going to require some sort of parse step whether the format is text or binary. That said, probably FlatBuffer would be even an optimized json parser, but json is crazy fast if the parser takes advantage of all modern processor optimizations.

There’s no way reading a series of text is going to beat a machine native byte representation, particularly for things like range limited integers. If you plan things right with proper word alignment, you don’t even have to copy the data to process it as you can directly dereference the byte offsets as machine native ints.

Re: Parsing gigabytes of JSON per second

#62

Post by morelisp »

heavenlyblue wrote: post #58
morelisp wrote: post #50

Earlier quoted context omitted.

Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.

But in order to semantically read protocol buffer files one needs to compile the schema files…

The degree you "need" to support the full schema format depends entirely on the language you're using. In Go you only need the annotated structures, or in Java you only need a mapping between field number and name, and the languages' own reflection capabilities can handle the rest.

Yes, strings, bytes, and substructures all appear in the same in the wire format. Just like strings, bytes, and dates all appear the same in JSON. If you're trying to write a generic protobuf viewer like wodenokoto suggests, you can make reasonable assumptions about the contents 99% of the time based on the data, and show the user multiple options if you're not sure. There are already lots of tools that do this, but none very well integrated into mainstream development workflows.

Re: Parsing gigabytes of JSON per second

#63

Post by sethev »

otrahuevada wrote: post #10

Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…

The use-case for a more efficient JSON parser is exactly the same as the use-case for a less efficient one. They're doing the same thing.

Daniel Lemire does a ton of performance research like this. He's an unusual case of a professor who publishes fully working source code.

There's a good chance that your database of choice uses some of his stuff or was influenced by his research.

Re: Parsing gigabytes of JSON per second

#64

Post by audray »

ec109685 wrote: post #59
tfsh wrote: post #53

Earlier quoted context omitted.

Protobufs would be a good contender here

Protobuf parsing is slower than an optimized json library, e.g. https://jsoniter.com/

I've been hacking with rust & wasm lately and am seeing positive results. Flat buffers are also promising if you are working with larger data.

Re: Parsing gigabytes of JSON per second

#65

Post by kevin_thibedeau »

EwanToo wrote: post #5
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON

Serialized binary encodings don't require a schema. They can have all the expressability of JSON and more.

Re: Parsing gigabytes of JSON per second

#66

Post by PeterisP »

secondcoming wrote: post #39
cogman10 wrote: post #14

Earlier quoted context omitted.

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

Indeed, out of all the trillions of HTTP requests that are communicated every day how many get read by a human. Not a lot, yet that's the use-case it's optimised for. Those on the cloud should compute how much HTTP headers count towards their traffic egress bill. Everyone is fine with human-readable as long as it's in English. Binary should be the default. If writing a binary to human decoder is too much for you, you…

Is a custom binary format that much more efficient than a gzipped human-readable format? If not, then it makes all sense to have a mostly universal decoder instead of a hundred different ones, especially since decoders of custom binary formats has traditionally been a major source of security vulnerabilities and other bugs even when those decoders have been written by skilled people and used in production for years.

Re: Parsing gigabytes of JSON per second

#67

Post by wilde »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

How do you debug? Human readability is valuable regardless of the size of the dataset.

Re: Parsing gigabytes of JSON per second

#68

Post by axiosgunnar »

throwawaygal7 wrote: post #24

This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…

> moderately sized (10-100TB/day)

weird flex but ok :-)

Re: Parsing gigabytes of JSON per second

#69

Post by pixl97 »

cogman10 wrote: post #28
AYBABTME wrote: post #21

Earlier quoted context omitted.

It's the same idea as designing hardware for repairability. Sure you can repair something that's welded on, but it's much easier if it's bolted through instead.

It's not the same. You are almost certainly passing these through tools (even built into the browser) that are doing extra processing to make it more readable. Let's assume, for example, CBOR ends up taking over JSON. Do you not think browsers wouldn't have a CBOR parser to make it more readable? This isn't bolt vs welding, this is bolt vs bolt with a washer. Yes there's a small extra step, but not some sort of insur…

The questions about readability when things go wrong. When binary data breaks, mostly your just screwed in figuring out went wrong, especially when it causes your decoder to fall over. When ASCII data breaks often a view of the data can point at something even when tools can't.

Re: Parsing gigabytes of JSON per second

#70

Post by rstuart4133 »

strictfp wrote: post #32
legulere wrote: post #27

Earlier quoted context omitted.

You mean like CBOR? https://en.wikipedia.org/wiki/CBOR

There's also Minecraft NBT. It's a bit obscure but quite nice. https://minecraft.fandom.com/wiki/NBT_format

Or indeed ASN.1 [0], which started life in 1984 and is still used in a lot of places today.

The idea has been around for almost 40 years in arguably more complete format than any of the above up to and including a standardised schema representation.

It's not the lack of a suitable standard that's holding it back.

[0] https://en.wikipedia.org/wiki/ASN.1

Post reply on HN