Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

81–90 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#81

Post by FpUser »

ericbarrett wrote: post #76
FpUser wrote: post #72

Earlier quoted context omitted.

I had exactly the same implementation except that type / version belonged to the whole message and would map to appropriate binary buffer in memory. No real de/serialization was needed. I still use it in my UDP game servers, with added packet id if message exceeds max datagram length and has to be split

The one concern I'd have with this format is a length field getting corrupted in transit and causing an out-of-bounds memory access. The network protocols' checksums won't save you 100% of the time, especially if there's bad hardware in the loop. If every field is fixed length this is less of a concern, of course; you might get bad data but you won't get e.g. a string with length 64M.

I do not remember it ever happening but being semi-paranoid I had length in 2 places - beginning and the end of the message.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#82

Post by joshlemer »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Doesn't it trivially have "sum types" since it's just arbitrary self-describing data? i.e. nobody is stopping you from passing around objects in such a way:

{a:1} {a:{b:2}} {a:4} {a:{b:4}}

There's no static type layer over top of this, so it's inherently up to interpretation and whatever type system you want to use to describe this data, to be able to express that the values of `a` can be of type `number | {b: number}`

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#83

Post by valenterry »

dastbe wrote: post #51
clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the…

That's not contradicting though.

One can always choose not to use (native) sumtypes if they are interested in extreme performance or compatibility.

But logically speaking, it is _good_ that it's a restriction that a sumtype can't just turn into a multiple-fields type. Because while my software (as the consumer) might still be able to deserialize it, the assumption that only one field is set would be broken and my logic would now potentially broken. Much better if that happens at deserialization time then later one when I find out that my data is incorrect/corrupt.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#84

Post by valenterry »

joshlemer wrote: post #82
clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Doesn't it trivially have "sum types" since it's just arbitrary self-describing data? i.e. nobody is stopping you from passing around objects in such a way: {a:1} {a:{b:2}} {a:4} {a:{b:4}} There's no static type layer over top of this, so it's inherently up to interpretation and whatever type system you want to use to describe this data, to be able to express that the values of `a` can be of type `number | {b: number…

> There's no static type layer over top of this

Yeah, that's the problem. I mean, hey, why json? We could just use unstructured plaintext for everything and now we are free to do everything. But obviously that has its own drawbacks.

Having built-in support for sumtypes means better and more ergonomic support from libraries, it means there is one standard and not different ways to encode things and it also means better performance and tooling.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#86

Post by n8ta »

the_girabbit wrote: post #70
n8ta wrote: post #40

I recently implemented a similar (simpler) format https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned -…

Ion will be even better for (structured) logging if this proposal for templates ever happens. https://github.com/amzn/ion-docs/pull/104 Looks like no one’s even so much as commented on it in the last year, so it might have been abandoned.

Ion is already a little too complex for my taste. It'd be a shame to see it go the same way as yaml where it's so complex that most major implementations are not safely interoperable.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#87

Post by kabdib »

jkhdigital wrote: post #79
hliyan wrote: post #57

Earlier quoted context omitted.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…

Classic example of a leaky abstraction, and the principle that implementation details inevitably become undocumented API behavior.

A lot of "transparent RPC" systems are like this. "It's just like a normal function call, it's sooo convenient" . . . until it isn't, because it involves the network hardware and configuration, routing environment, firewalls, equipment failure . . .

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#88

Post by mendigou »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

This is exactly how it's done for spacecraft telemetry and telecommand too, but in this case it's to save bytes rather than processing time.

I also miss working on those systems.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#89

Post by jsolson »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Protobuf supports sum types in the higher-level generated descriptors and languages -- on the wire they're just encoded as, well... oneof a number of possible options.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#90

Post by NavinF »

armchairhacker wrote: post #68
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

You just described protobufs and all its successors.

See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html

It’s a 64bit random number so it’ll never have unintentional collisions.

Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

Post reply on HN