Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JCOF: JSON-like Compact Object Format

github.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: JCOF: JSON-like Compact Object Format

#51

Post by zimpenfish »

Beltiras wrote: post #43
zimpenfish wrote: post #25

Earlier quoted context omitted.

> To be fair you would gzip the JCOF encoding in this example too. Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?

10% is a lot of your egress.

A fair point.

Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k.

If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)

Re: JCOF: JSON-like Compact Object Format

#52

Post by zimpenfish »

Beltiras wrote: post #48
masklinn wrote: post #30

Earlier quoted context omitted.

No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> A sibling has the data.

And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!

Re: JCOF: JSON-like Compact Object Format

#53

Post by mike_hock »

conradev wrote: post #28
xsmasher wrote: post #20

Earlier quoted context omitted.

comets.json: JSON: 51949 bytes JCOF: 37480 bytes (0.721x) JSON ZIPED: 15178 bytes Zipped json wins again.

Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses

And zipped/unzipped msgpack for that matter.

Re: JCOF: JSON-like Compact Object Format

#54

Post by indymike »

nine_k wrote: post #44
indymike wrote: post #18

JCOF is not usable in many of the use cases where JSON is. Here is why: * JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines. * Nested objects look to be difficult, especially if the schema is not…

JCOF is just not radical enough. It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc. BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.

> It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.

There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serializati...

> BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.

Sounds like XML, which is a whole other discussion :-)

Re: JCOF: JSON-like Compact Object Format

#55

Post by twawaaay »

conradev wrote: post #50
bearjaws wrote: post #36

Earlier quoted context omitted.

It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip. If it could somehow produce a significant reduction with fewer CPU cycles, ther…

Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression

In my experience it should not make much or any difference.

I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.

Re: JCOF: JSON-like Compact Object Format

#56

Post by SahAssar »

mywittyname wrote: post #31
andrew_ wrote: post #12

Earlier quoted context omitted.

Nevermind readable

I'm still not sure wtf the leading 0, on every object is supposed to represent.

I think that's representing the keys, so if you'd have multiple types of objects in an array the key table would have multiple rows and the leading number index for each object would define what the keys for that object would be.

Re: JCOF: JSON-like Compact Object Format

#57

Post by zimpenfish »

zimpenfish wrote: post #52
Beltiras wrote: post #48

Earlier quoted context omitted.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> A sibling has the data. And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!

EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.

Re: JCOF: JSON-like Compact Object Format

#58

Post by zimpenfish »

zimpenfish wrote: post #51
Beltiras wrote: post #43

Earlier quoted context omitted.

10% is a lot of your egress.

A fair point. Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k. If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)

EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.

Re: JCOF: JSON-like Compact Object Format

#59

Post by smitty1e »

LAC-Tech wrote: post #41

The comments here are really negative. I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome. And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.

If a specific use-case can benefit from it, fine.

I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.

Re: JCOF: JSON-like Compact Object Format

#60

Post by beepbooptheory »

jsiaajdsdaa wrote: post #21
tgv wrote: post #16

Earlier quoted context omitted.

While Oscar Wilde would undoubtedly have something witty to say about that, I guess it's a matter of stress: if you read it as a word and assume stress on the (invisible) first vowel, it's almost inevitable: juh-cof.

But we dont say juh sahn for JSON? Hence why Jcough seemed intuitive.

Guess you could say if its a 'jack off' here, its 'jiss on' there.
Post reply on HN