Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

171–180 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#171

Post by lordnacho »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Same here, I wrote an exchange core that did this using SBE. Basically you don't serialize in the classical sense, because you're simply taking whatever bytes are at your pointer and using them as some natural type. The internals of the exchange also simply used the same layout, so there was minimal copying and interpreting. On the way out it was the same, all you had to do was mask a few fields that you didn't want everyone to see and ship it onto the network.

Even an unoptimized version of this managed to get throughput in the 300K/s range.

Somehow it's the endpoint of my journey into serialization. Basically, avoid it if you need to be super fast. For most things though, it's useful to have something that you can read by eye, so if you're not in that HFT bracket it might be nicer to just use JSON or whatever.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#172

Post by gpderetta »

lozenge wrote: post #144
elcritch wrote: post #136

Earlier quoted context omitted.

It does apply, according to https://www.johndcook.com/blog/2017/01/10/probability-of-sec...

But if my application only uses 100 schemas, I only care about a collision if it's with one of those 100.

You have a collision if any two schemas share the id, not if a specific schema collides with any of the others. So it is exactly like the birthday paradox.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#174

Post by ImprobableTruth »

joshlemer wrote: post #113
valenterry wrote: post #112

Earlier quoted context omitted.

No, that is not a sumtype, that's an array. In the same sense "1e-12" is not a number, it's a string. Yes, it's a string that encodes a number in a certain notion, but for alle the tooling, the IDE, the libraries, etc. it will stay a string.

What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list `

Sum types =/= union types. Sum types are also called 'tagged' or 'discriminable' unions because they have some way to discriminate between them. That is, if you have an element a of type A, a is not part of the sum type A + B because it's missing a tag.

[5,"hello",3] has the type list (int ∪ string), not list (int + string). You can emulate the latter by manually adding a tag, but native support is much preferable.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#175

Post by makotobestgirl »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Sounds like Google's flatbuffers [0], which indexes directly into a byte buffer using the field size prefix.

[0] https://google.github.io/flatbuffers/

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#176

Post by ricardobeat »

jsolson wrote: post #89
clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

Protobuf supports sum types in the higher-level generated descriptors and languages -- on the wire they're just encoded as, well... oneof a number of possible options.

Which results in very painful inconsistencies when you’re dealing with the same schema on different platforms.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#177

Post by kwertyoowiyop »

angstrom wrote: post #52
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

Every multiplayer game programmer from the 1990s agrees with you!

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#178

Post by heavenlyblue »

garmaine wrote: post #93
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

I don’t understand your maths here: how is generating 4billion of them is any different from generating 3 billion except a slight raise in the probability measure?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#179

Post by kwertyoowiyop »

silvestrov wrote: post #135

This is what JSON should have been extended to. But Douglas Crockford just don't want to innovate anything, just like Gruber didn't want to make a proper specification of the Markdown format. Sometimes people are keeping innovation back. Fortunately this did not happend with html. The main thing missing from the text format is a magic and version number. At least the binary format has it.

The dominance of JSON shows that Crockford made some good decisions, even though we may not agree with them on any given day.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#180

Post by heavenlyblue »

garmaine wrote: post #139
NavinF wrote: post #100

Earlier quoted context omitted.

Yes it is. Message schemas are made by humans. Most of these messages will be extended in a backwards compatible manner over the life of a project rather than replaced entirely so their IDs don’t change. That’s kinda the point of protobufs and its successors. I’ve probably generated 100 IDs over my lifetime.

Which puts it on the same order of magnitude as the number of people on the planet. If every person alive generated a schema (or if 1/100th of all people generate 100 IDs each like you) then we'd have a small number of collisions. More likely you'd get large numbers of schema like that if there's a widespread application of a protocol compiler that generates new schema programmatically, e.g. to achieve domain separat…

If you have 4 billion of them generated there’s another 1/4billionth chance you’ll generate a duplicate.

On top of that you would not only need to generate the same ID, you would need to USE it in the same system where that is could have some semantics to not cause an error.

Post reply on HN