Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing gigabytes of JSON per second

arxiv.org

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Parsing gigabytes of JSON per second

#71

Post by ec109685 »

koolba wrote: post #61
ec109685 wrote: post #60

Earlier quoted context omitted.

Why do you think human readable adds appreciable overhead? If you want to create a flexible interchange format, that is going to require some sort of parse step whether the format is text or binary. That said, probably FlatBuffer would be even an optimized json parser, but json is crazy fast if the parser takes advantage of all modern processor optimizations.

There’s no way reading a series of text is going to beat a machine native byte representation, particularly for things like range limited integers. If you plan things right with proper word alignment, you don’t even have to copy the data to process it as you can directly dereference the byte offsets as machine native ints.

But that isn’t an interchange format given different processors represent such things differently.

Re: Parsing gigabytes of JSON per second

#72

Post by manquer »

cogman10 wrote: post #14
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…

The human readability has to do with developer tooling and ability to work that format. Abstractions are all optimized towards developer productivity the costliest part not compute cycles.

Re: Parsing gigabytes of JSON per second

#73

Post by Too »

matja wrote: post #46
wodenokoto wrote: post #33

Earlier quoted context omitted.

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

Tricky part about implementing that is that the protobuf wire-format alone doesn't contain enough information to unambiguously represent the real data types, it also needs the schema (.proto file) to do that. For example, in the wire-format, a string and a sub-message are encoded as the same type (a blob - varint + sequence of bytes), but using the schema they are clearly interpreted differently. Sure, it is possible…

Json schema has a similar challenge, there is no header, in the json files themselves, indicating which schema is used.

The slightly dirty solution is https://www.schemastore.org/json/, where the IDE looks up schema from a global registry, using a fileMatch pattern.

Re: Parsing gigabytes of JSON per second

#74

Post by rixed »

stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

"we as an industry" hurts.

It's funny how I feel questioning JSON here on HN is like starting a discussion about politics in a family diner.

I'm forever grateful to Google for demonstrating that all "the industry" is not fully committed to HTTP, JSON and scripting languages. It's honestly a relief to have encountered some sanity somewhere.

Oh god please help me, I just did it, I questioned JSON on HN!

Re: Parsing gigabytes of JSON per second

#75

Post by 7952 »

SigmundA wrote: post #22
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

Although it's not a format, the closest thing I have found is Postgres. You get well defined types that make life easier. They are stored efficiently in binary. And have a sensible text representation that means you can still read and write them by hand. I can store a PostGIS geometry with millions of verticies and still use text to create a point or read an envelope. Text formatting is hugely useful and benefits from standardisation. But that does not need to be built into the format.

Re: Parsing gigabytes of JSON per second

#76

Post by rytcio »

pixl97 wrote: post #69
cogman10 wrote: post #28

Earlier quoted context omitted.

It's not the same. You are almost certainly passing these through tools (even built into the browser) that are doing extra processing to make it more readable. Let's assume, for example, CBOR ends up taking over JSON. Do you not think browsers wouldn't have a CBOR parser to make it more readable? This isn't bolt vs welding, this is bolt vs bolt with a washer. Yes there's a small extra step, but not some sort of insur…

The questions about readability when things go wrong. When binary data breaks, mostly your just screwed in figuring out went wrong, especially when it causes your decoder to fall over. When ASCII data breaks often a view of the data can point at something even when tools can't.

Reading a binary file is no different than reading ascii. I experimented with writing a debugger, which meant learning how to parse executable ELF binaries. It really wasn't bad at all. It just takes a bit more time to figure out what the bytes/data are expect at what offsets. The real un-said reason is a) programmers are lazy b) its easier/cheaper to hire people who can read ascii than find someone who is a bit more experienced and can figure out how to debug binary parsing.

Re: Parsing gigabytes of JSON per second

#77

Post by rytcio »

PeterisP wrote: post #66
secondcoming wrote: post #39

Earlier quoted context omitted.

Indeed, out of all the trillions of HTTP requests that are communicated every day how many get read by a human. Not a lot, yet that's the use-case it's optimised for. Those on the cloud should compute how much HTTP headers count towards their traffic egress bill. Everyone is fine with human-readable as long as it's in English. Binary should be the default. If writing a binary to human decoder is too much for you, you…

Is a custom binary format that much more efficient than a gzipped human-readable format? If not, then it makes all sense to have a mostly universal decoder instead of a hundred different ones, especially since decoders of custom binary formats has traditionally been a major source of security vulnerabilities and other bugs even when those decoders have been written by skilled people and used in production for years.

The answer is in your comment. If you add a step of decompression, then text parsing...it's plain-as-day going to be a lot slower then just pushing bits through a binary parser. Less work = faster

Re: Parsing gigabytes of JSON per second

#78

Post by cogman10 »

rytcio wrote: post #76
pixl97 wrote: post #69

Earlier quoted context omitted.

The questions about readability when things go wrong. When binary data breaks, mostly your just screwed in figuring out went wrong, especially when it causes your decoder to fall over. When ASCII data breaks often a view of the data can point at something even when tools can't.

Reading a binary file is no different than reading ascii. I experimented with writing a debugger, which meant learning how to parse executable ELF binaries. It really wasn't bad at all. It just takes a bit more time to figure out what the bytes/data are expect at what offsets. The real un-said reason is a) programmers are lazy b) its easier/cheaper to hire people who can read ascii than find someone who is a bit more…

Yeah and to b, it really just comes down to a lack of tools and people experienced in those tools (IMO).

There's no reason a binary format couldn't be as easy to read as a JSON format. All you'd really need is a "binary->json" gui and you're off.

It would take a little effort to make such a tool and you could have it integrated into every browser.

It's not like something like that is unprecedented even, After all, there's no browser tool out there that's showing you the gzipped resource on a compressed endpoint. It's always already doing the step of gunzipping. binary->Json would similarly be just an additional step the browser could do in the tools.

Re: Parsing gigabytes of JSON per second

#79

Post by sdfdf4434r34r »

SigmundA wrote: post #22
stkdump wrote: post #4

If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

>{ contentType: "image/jpeg", content: }

It would have to be encoded somehow, because what if the "real binary data" included the byte 0x7D which is '}'

Re: Parsing gigabytes of JSON per second

#80

Post by SigmundA »

sdfdf4434r34r wrote: post #79
SigmundA wrote: post #22

Earlier quoted context omitted.

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON. I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc). Text file…

>{ contentType: "image/jpeg", content: } It would have to be encoded somehow, because what if the "real binary data" included the byte 0x7D which is '}'

Sorry thats not meant to be a literal format but conceptually mapped to text JSON. It would instead be more like CBOR or other JSON like binary formats at the byte level.
Post reply on HN