Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JCOF: JSON-like Compact Object Format

github.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: JCOF: JSON-like Compact Object Format

#61

Post by mort96 »

indymike wrote: post #18

JCOF is not usable in many of the use cases where JSON is. Here is why: * JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines. * Nested objects look to be difficult, especially if the schema is not…

Hey, JCOF author here. I didn't notice this had made it to HN before now.

* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.

* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).

* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.

I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.

Re: JCOF: JSON-like Compact Object Format

#62

Post by mort96 »

dathinab wrote: post #39

no offense, but this seems pointless tbh. If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support. or just use a compression (after which the difference between JSON and JCOF should be negligible)

I'm curious, which more compact format would you use? Because I have looked a bit and been unable to find one, both MessgePack and CBOR produce pretty large objects. CBOR with the string reference extension comes closer, but it's still bigger, and string references don't seem very well supported in the CBOR ecosystem.

Do you mean you'd use a format with a schema?

Re: JCOF: JSON-like Compact Object Format

#63

Post by dreamdeath »

twawaaay wrote: post #55
conradev wrote: post #50

Earlier quoted context omitted.

Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression

In my experience it should not make much or any difference. I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size…

I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).

Re: JCOF: JSON-like Compact Object Format

#64

Post by HelloNurse »

jsiaajdsdaa wrote: post #15
HelloNurse wrote: post #11

Earlier quoted context omitted.

Jack off, I'm afraid.

Interesting, I wonder which type of English speakers would see that as their initial hunch for pronunciation.

There is a very, very small difference between placing a slight break in the middle of the acronym as J-COF or as JC-OF, because there's a major unwritten vowel sound after J in any case.

Re: JCOF: JSON-like Compact Object Format

#65

Post by indymike »

mort96 wrote: post #61
indymike wrote: post #18

JCOF is not usable in many of the use cases where JSON is. Here is why: * JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines. * Nested objects look to be difficult, especially if the schema is not…

Hey, JCOF author here. I didn't notice this had made it to HN before now. * JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those st…

Oh, I think you have something, but I don't think JSON and JCOF are comparable. I think it makes a very interesting replacement for formats where there is a header line like CSV, TSV. JCOF allows for some level of consistent nested structures (which would be nice for header/detail records which are really common in the real world) and would be so much nicer than many of the bespoke formats you see out there... and also, sooo much more efficient than XML, and without the issues that come with XML. Incidentally, these kinds of uses are where JSON really doesn't work well because you always have the potential of a mis-shaped object.

In short: nice work. Might be fun to roll an implementation or two :-)

Re: JCOF: JSON-like Compact Object Format

#66

Post by KronisLV »

Beltiras wrote: post #48
masklinn wrote: post #30

Earlier quoted context omitted.

No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.

Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.

Re: JCOF: JSON-like Compact Object Format

#67

Post by Beltiras »

KronisLV wrote: post #66
Beltiras wrote: post #48

Earlier quoted context omitted.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementa…

The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.

Re: JCOF: JSON-like Compact Object Format

#68

Post by petre »

mike_hock wrote: post #53
conradev wrote: post #28

Earlier quoted context omitted.

Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses

And zipped/unzipped msgpack for that matter.

Precisely, I'd rather use msgpack or CBOR if it requires additional libs. I use JSON because of browser support.

Re: JCOF: JSON-like Compact Object Format

#69

Post by twawaaay »

dreamdeath wrote: post #63
twawaaay wrote: post #55

Earlier quoted context omitted.

In my experience it should not make much or any difference. I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size…

I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).

Well, that's true. But if performance is more important to you, just skip it and go for binary formats that can be orders of magnitude more efficient.

In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.

If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.

Post reply on HN