Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

61–70 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#61

Post by vlovich123 »

dastbe wrote: post #51
clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the…

Have you looked at cap’n’proto. It does sum types in a very sane way.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#63

Post by vlovich123 »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Cap’n’proto has native sum types.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#65

Post by whimsicalism »

oandrew wrote: post #23

So basically it's Amazon's version of Apache Avro. Avro supports binary/json serialization, schema evolution , logical types (e.g. timestamp) and other cool stuff. https://avro.apache.org/docs/current/spec.html

... or thrift ... or protobuf

https://xkcd.com/927/

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#66

Post by hackcasual »

Zamicol wrote: post #59

Am I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/

You want to use hex whenever byte aligned data is going to be compressed. Base64 quadruples byte level symbols

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#67

Post by ralusek »

Zamicol wrote: post #59

Am I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/

I understand the case for Base91, but why hex over Base64? Base64 for readability and sticking to multiples of two, Base91 for maximum efficiency with readable ASCII.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#68

Post by armchairhacker »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

I don't understand why serialization formats that separate structure and content aren't more popular.

Imagine a system every message is a UID or DID (https://www.w3.org/TR/did-core/) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about.

Once a client learns a UID, messages are about as compact as possible. And the data defining UIDs can be much more descriptive than e.g. property names in JSON. You can send documentation and other excess data when defining the UID, because you don't have to worry about size, because you're only sending the UID once. And UIDs can reference other UIDs to reduce duplication.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#69

Post by Aeolun »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key.

If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#70

Post by the_girabbit »

n8ta wrote: post #40

I recently implemented a similar (simpler) format https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned -…

Ion will be even better for (structured) logging if this proposal for templates ever happens. https://github.com/amzn/ion-docs/pull/104

Looks like no one’s even so much as commented on it in the last year, so it might have been abandoned.

Post reply on HN