51–60 of 81 posts
Earlier quoted context omitted.
You mean like CBOR? https://en.wikipedia.org/wiki/CBOR
There's also Minecraft NBT. It's a bit obscure but quite nice. https://minecraft.fandom.com/wiki/NBT_format
Having some trouble figuring out how to benefit from this. I can't think of a scenario where JSON handling is our bottleneck; almost all of the massive-data-handling tasks in my entire career have usually been handled by our database of choice. Largest individual JSONs we've had to handle like on import tasks and such were about 300mb, which proved extremely easy to manage with stuff like JSONStream -I think we used…
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
Earlier quoted context omitted.
I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
That’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
I've gotta say, I don't really understand why the industry is so in love with human readability for computer to computer interaction. It seems like a notion that started in the early internet and just refuses to die. Everything on the internet is minified and compressed at this point, so the entire idea of "human readability" left the station years ago. Yet I'll still see a primary complaint against the likes of http…
The project described in the paper is https://simdjson.org/ .
PyPI: https://pypi.org/project/pysimdjson/
There's a rust port: https://github.com/simd-lite/simd-json
... From ijson https://pypi.org/project/ijson/#id3 which supports streaming JSON:
> Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends: [yajl2_c, yajl2_cffi, yajl2, yajl, python]
Earlier quoted context omitted.
IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...
Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
Protobufs would be a good contender here
If your dataset is large enough to benefit from such a hyperoptimized parser you might not benefit from the human readability anymore, which is the main reason for the required CPU cycles on the first place. So you should probably use a feature-equivalent binary format that is optimized for parsing speed. The only reason to use json then is that it is the lingua franca. So we as an industry should figure out which of…
That said, probably FlatBuffer would be even an optimized json parser, but json is crazy fast if the parser takes advantage of all modern processor optimizations.