21–30 of 69 posts
Earlier quoted context omitted.
Interesting, I wonder which type of English speakers would see that as their initial hunch for pronunciation.
While Oscar Wilde would undoubtedly have something witty to say about that, I guess it's a matter of stress: if you read it as a word and assume stress on the (invisible) first vowel, it's almost inevitable: juh-cof.
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.
In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.
Earlier quoted context omitted.
curl https://raw.githubusercontent.com/mortie/jcof/main/tests/corpus/meteorites.json | gzip -9 | wc Gives me 34569 So the comparison is: JSON: 244920 bytes JCOF: 87028 bytes GZIP: 34569 bytes
To be fair you would gzip the JCOF encoding in this example too. Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for. If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
Earlier quoted context omitted.
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea,…
Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings. This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
You're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
For those unfamiliar, here is the interesting bit:
> rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
comets.json: JSON: 51949 bytes JCOF: 37480 bytes (0.721x) JSON ZIPED: 15178 bytes Zipped json wins again.
Earlier quoted context omitted.
You're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
Depends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ is what i've been toying with recently. For those unfamiliar, here is the interesting bit: > rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no…
It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends
It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes
Earlier quoted context omitted.
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M. But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M. In summary, neat idea,…
JCOF and gzip are not mutex.