Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

91–100 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#91

Post by infogulch »

NavinF wrote: post #90
armchairhacker wrote: post #68

Earlier quoted context omitted.

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

Interesting. Ids in particular are described here: https://capnproto.org/language.html#unique-ids

I wonder if giving it a name based on the hash of the definition has been explored; like Unison [0] where all code is content addressable, but for just capnproto definitions. Is there a reason not to?

[0]: https://www.unisonweb.org

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#92

Post by boxfire »

NavinF wrote: post #90
armchairhacker wrote: post #68

Earlier quoted context omitted.

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#93

Post by garmaine »

NavinF wrote: post #90
armchairhacker wrote: post #68

Earlier quoted context omitted.

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

> It’s a 64bit random number so it’ll never have unintentional collisions.

It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#94

Post by danachow »

Aeolun wrote: post #69
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?

OT but keyboard latency can and often is far below 50ms, more like 1ms. It seems to be a common misconception that denouncing mandates increased lag.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#95

Post by remram »

boxfire wrote: post #92
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically

MD5 is safe against unintentional collisions.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#96

Post by remram »

garmaine wrote: post #93
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

When you reach the 4 billionth version of your protocol?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#97

Post by NavinF »

boxfire wrote: post #92
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically

Dude that’s why I said “unintentional collisions”.

Of course you can get intentional collisions. The security model here assumes that anyone that wants to know your message’s ID can just ask.

Did you know that the Internet Protocol uses a 4-bit header to specify the format (v4 or v6) of the rest of the message? They should have used 128 bits. What a bunch of fools.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#98

Post by NavinF »

infogulch wrote: post #91
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

Interesting. Ids in particular are described here: https://capnproto.org/language.html#unique-ids I wonder if giving it a name based on the hash of the definition has been explored; like Unison [0] where all code is content addressable, but for just capnproto definitions. Is there a reason not to? [0]: https://www.unisonweb.org

Capnp uses the name of your message, but not its full definition because that would make it impossible to extend protocols in a backwards compatible way. Without the ability to add new fields, making changes to your protocol would be impossible in large orgs.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#99

Post by angstrom »

hliyan wrote: post #57
angstrom wrote: post #52

Earlier quoted context omitted.

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…

Yeah, that's part of the trick for large listing responses to be spread across frames. Usually with some indicator like a "more flag" so the client can say "get me the next sequence by requesting the next index in the listup with the prior btree index. People do this all the time with large databases and it's a very similar use case.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#100

Post by NavinF »

garmaine wrote: post #93
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

Yes it is. Message schemas are made by humans. Most of these messages will be extended in a backwards compatible manner over the life of a project rather than replaced entirely so their IDs don’t change. That’s kinda the point of protobufs and its successors.

I’ve probably generated 100 IDs over my lifetime.

Post reply on HN