Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

31–40 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#33

Post by setheron »

Wow I remember using Ion back at Amazon in 2012. I can’t remember but I think the order data warehouse was using it …

I also now remember back to using something that was akin to FaaS but wasn’t called that. I could give them a JAR of some code that would execute on some Ion data for the order data when it changed. Basically FaaS for an ETL pipeline…

Crazy how ahead of the times some companies were.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#34

Post by fnord77 »

oandrew wrote: post #23

So basically it's Amazon's version of Apache Avro. Avro supports binary/json serialization, schema evolution , logical types (e.g. timestamp) and other cool stuff. https://avro.apache.org/docs/current/spec.html

I wanted to see what the differences are between Ion and Avro.

Unlike avro, ion doesn't require a schema.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#35

Post by vineyardmike »

setheron wrote: post #33

Wow I remember using Ion back at Amazon in 2012. I can’t remember but I think the order data warehouse was using it … I also now remember back to using something that was akin to FaaS but wasn’t called that. I could give them a JAR of some code that would execute on some Ion data for the order data when it changed. Basically FaaS for an ETL pipeline… Crazy how ahead of the times some companies were.

I wonder why it took 10+ years to share then?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#36

Post by OJFord »

dorianmariefr wrote: post #6

Pretty neat, but isn't it like *two* formats: one binary and one textual?

Consider that binary, binary coded decimal, Gray code, hexadecimal, octal, etc. are all 'formats' expressing the same (numerical) idea.

You can't say the same of, for example, YAML & JSON, since the former (if not the latter?) has constructs unrepresentable in the other.

It's slightly confused because an application might 'serialise to' JSON or YAML or Ion equivalently - but really that's saying the application's data being serialised fits a model that's a subset of the intersection between those formats.

You could call Ion two, but it's more than that in that it's also a promise that they're 1:1 (err, and onto if you like) - their intersection is their union.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#38

Post by clhodapp »

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#39

Post by timdorr »

vineyardmike wrote: post #35
setheron wrote: post #33

Wow I remember using Ion back at Amazon in 2012. I can’t remember but I think the order data warehouse was using it … I also now remember back to using something that was akin to FaaS but wasn’t called that. I could give them a JAR of some code that would execute on some Ion data for the order data when it changed. Basically FaaS for an ETL pipeline… Crazy how ahead of the times some companies were.

I wonder why it took 10+ years to share then?

Actually, it only took them 4-ish years: https://amzn.github.io/ion-docs/news/2016/04/21/amazon-open-...

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#40

Post by n8ta »

I recently implemented a similar (simpler) format https://baremessages.org/ in ruby.

First thoughts are:

ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare

ION cons - what's the text format even for?

BARE pros: - schemas keep things tightly versioned - smaller binaries (not self describing like ion) - simpler to implement so tons of devs have impl'ed for their favorite lang - better suited to small messages (think REST json api)

BARE cons: - no skip read - no date support

I might do an ion ruby implementation too, to really feel out the difference.

Post reply on HN