Amazon Ion Specification
amazon-ion.github.io
Amazon Ion Specification
1–10 of 38 posts
Re: Amazon Ion Specification
#2Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Re: Amazon Ion Specification
#3On the one hand, this seems more real than a lot of promotion-ware I've seen: https://github.com/amazon-ion/ion-intellij-plugin#readme
On the other hand, they're not using this for the boto schemas, which seems like a natural place to show that it's able to capture real-world schemas so that makes it hard for me to think this has any traction
Re: Amazon Ion Specification
#4Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Why do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization
Re: Amazon Ion Specification
#5Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Why do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization
I work in an a shop where Kafka and Avro are everywhere. If I worked someone else I might make reference to something else if it was front-of-mind all the time.
Re: Amazon Ion Specification
#6On the one hand, this seems more real than a lot of promotion-ware I've seen: https://github.com/amazon-ion/ion-intellij-plugin#readme On the other hand, they're not using this for the boto schemas, which seems like a natural place to show that it's able to capture real-world schemas so that makes it hard for me to think this has any traction
Ion is heavily used on the retail side of Amazon, but it's only recently started to appear in AWS products.
AWS is starting support PartiQL (https://partiql.org/) queries in some places and PartiQL uses Ion's type system internally.
Re: Amazon Ion Specification
#7On the one hand, this seems more real than a lot of promotion-ware I've seen: https://github.com/amazon-ion/ion-intellij-plugin#readme On the other hand, they're not using this for the boto schemas, which seems like a natural place to show that it's able to capture real-world schemas so that makes it hard for me to think this has any traction
The SDKs use Smithy[1] which is tailored for defining+generating services and SDKs, Ion is more of a pure data serialization format. It's definitely niche but my org uses it in a few places and it has some nice properties that fit our case pretty well (rapidly evolving schema, most clients only care about a small subset of attributes, ability to apply multiple and different schemas based on regions or businesses).
It's the sort of thing where I'd advise exploring other options first and only using it if the whys[2] really resonate with you because it definitely comes with some overhead.
[1] https://smithy.io/2.0/index.html [2] https://amazon-ion.github.io/ion-docs/guides/why.html
Re: Amazon Ion Specification
#8Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Ion has a binary format but is not specifically about real-time streaming. It is a JSON replacement.
Ion originated 10+ years ago from the Amazon catalog team - the team that kept data about the hundreds of millions of items available on Amazon. Nearly every team in the company called the catalog to get information about items all the time - scanning the entire catalog, parts of the catalog, millions of individual item lookups every second, etc.
They did the math and some very large percentage of network traffic in Amazon Retails data centers was catalog data. If that data, currently in XML or JSON format, was sent in a more compact format it would save some ridiculous millions of dollars every year. So Ion was born and eventually open sourced.
Re: Amazon Ion Specification
#9Previous discussions on Ion:
https://news.ycombinator.com/item?id=29284428 (2 years ago, 229 comments)
https://news.ycombinator.com/item?id=23921610 (3 years ago, 110 comments)
https://news.ycombinator.com/item?id=11546098 (7 years ago, 163 comments)
Re: Amazon Ion Specification
#10Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Why do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization
People keep inventing new ones because the old ones suck or they think the old ones suck. Look at all the discontents around JSON (no comments!), people react violently when people try to apply a little extra like JSON-LD. Then there are all the things like YAML, TOML and such that try to be a little better but are widely thought to be a little worse. (And that's just the human readable data formats) Then there is always
https://en.wikipedia.org/wiki/ASN.1
which is forgotten but not gone.