41–50 of 69 posts
I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome.
And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.
Thing about JSON is the ease of reading and writing it, regardless the size of the document. Compression deals with the rest.
For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)
Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.
It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.
Requiring parsers to keep keys in file order also might have been a good thing.
(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
Earlier quoted context omitted.
To be fair you would gzip the JCOF encoding in this example too. Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for. If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
> To be fair you would gzip the JCOF encoding in this example too. Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
JCOF is not usable in many of the use cases where JSON is. Here is why: * JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines. * Nested objects look to be difficult, especially if the schema is not…
It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
Earlier quoted context omitted.
JCOF and gzip are not mutex.
You just blew my mind that “mutex” is a portmanteau of “mutually exclusive.” How have I been programming for 20 years and never realized this?
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
OTOH zstd [1] should be significantly more efficient, while being as fast or faster.
Earlier quoted context omitted.
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
Earlier quoted context omitted.
JCOF and gzip are not mutex.
No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
JSON isn't mean to be efficient, but to be a good compromise between being human readable/writable, easily generated/parsed by a machine, and familiar. CBOR and BSON make more sense, to be frank, for any use case I could think of using this in.
We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.
This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
Earlier quoted context omitted.
Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses
It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip. If it could somehow produce a significant reduction with fewer CPU cycles, ther…