(2016)
Amazon Ion Specification
31–38 of 38 posts
Re: Amazon Ion Specification
#32lol, the internal docs on this at Amazon were something very close to "we invented this before Avro and we think that's probably a better choice if you need binary serialization." My 2 cents: don't use it.
[dead]
Re: Amazon Ion Specification
#33I would be interested to see how this compares to something like msgpack [1] in performance and final size of the binary. Msgpack has been my go-to for binary serialization for years due to how simple and fast it is, and how easy it is to make it work with native Clojure data structures. [1] https://msgpack.org/index.html
That comparison would depend heavily on what you're storing. Ion has the option of using symbol tables to replace strings (e.g. in struct/map keys or in values). So, if you benchmark had a large number of records with similar structures, I would expect Ion to pull ahead. On the other hand, if each record had nothing in common, I'd expect them to perform similarly. One feature of the Ion libraries that I've liked is t…
Symbol tables, compression, etc seem one level of abstraction above what msgpack provides. Such features could be implemented on top of vanilla msgpack as long as all parties agree on the msgpack schema.
Re: Amazon Ion Specification
#34What's the pros and cons of this versus CBOR, which we had great success with in our system. https://cbor.io
Pros of Ion vs CBOR:
Wider range of data types - Ion supports decimals, symbols, blobs, and clobs which don't exist in CBOR. Optional schemas and annotations - Ion allows attaching type/schema information to data for validation purposes. CBOR has no schema support. Text format - Ion provides a human-readable text format for data interchange, CBOR is binary only. Maturity - Ion has been used in production at Amazon since 2009, CBOR is a newer standard (RFC 7049 in 2014). Language support - More mature library ecosystem around Ion vs CBOR which is still gaining adoption.
Pros of CBOR vs Ion:
Standardized - CBOR is an IETF standard, Ion is an Amazon-proprietary format. Simplicity - CBOR has a smaller set of basic data types making it simpler to implement. Used in other standards - CBOR is used in data formats like COSE for crypto operations and CWT for web tokens. Efficiency - The CBOR binary format can have a smaller encoding size than Ion's. JSON interoperability - CBOR is designed to be a JSON-compatible binary format. Ion is JSON-like but not fully compatible.
In summary, Ion has richer data typing and schema capabilities and a long production history. But CBOR is simpler, standardized, and gaining momentum - especially in crypto and web standards using it as a binary encoding basis.
So Ion may be better for applications dealing with complex, annotated data. But CBOR has advantages for an efficient binary interchange format, particularly when standards compatibility is important.
Re: Amazon Ion Specification
#35Earlier quoted context omitted.
My 2 cents: don't use avro or anything like it unless you can prove its going to save you money
What would you suggest? Just JSON everywhere?
My $0.02 is "yes, JSON everywhere". Specifically, one object per line, newline delimited, sorted keys, compressed with zstd or gzip.
Re: Amazon Ion Specification
#36I would be interested to see how this compares to something like msgpack [1] in performance and final size of the binary. Msgpack has been my go-to for binary serialization for years due to how simple and fast it is, and how easy it is to make it work with native Clojure data structures. [1] https://msgpack.org/index.html
MessagePack goes to "extremes" to shrink message size; I suspect that at least to win out.
Re: Amazon Ion Specification
#37Re: Amazon Ion Specification
#38Earlier quoted context omitted.
Why do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization
People keep inventing new ones because the old ones suck or they think the old ones suck. Look at all the discontents around JSON (no comments!), people react violently when people try to apply a little extra like JSON-LD. Then there are all the things like YAML, TOML and such that try to be a little better but are widely thought to be a little worse. (And that's just the human readable data formats) Then there is al…
Oh man ASN.1, makes me shiver, oh the memories…
I hadn’t thought about it in like a decade but yeah it’s still silently in the background…