Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

191–200 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#191

Earlier quoted context omitted.

This is protocol buffers + a global type registry. I worked on such a system.

Is it public? Id love to learn more about it.

If you read the protobuf source, you can see a bunch of places where you can hook in custom type-fetching code, e.g. in the google.protobuf.Any type.

After studying it a bit, I'm certain this is how it's used inside Google (might also be mentioned elsewhere).

All you'd really need to do is to compile all protos into a repository (you can spit out the binary descriptors from protoc), then fetch those and decode in the client.

It'd actually be quite straightforward to set up,

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#192
post #69

Earlier quoted context omitted.

> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?

OT but keyboard latency can and often is far below 50ms, more like 1ms. It seems to be a common misconception that denouncing mandates increased lag.

That's because a lot of input hardware uses moronic debouncing.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#193
post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Very neat and similar to a project I am starting for packet radio. I went further with the dictionary concept so that it contains common data. This way, your message contains only a few dictionary "pointers" (integers in base 64). This makes it easier to fit messages in ASCII for 300 baud links.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#194

Check out Ilya Yaroshenko’s Ion library for D, part of the larger ‘mir’ library: http://mir-ion.libmir.org/ https://github.com/libmir/mir-ion

Weird to see the library I work on show up in HN —- Mir Ion is a pretty complicated library (and admittedly our documentation needs work — I’m working on that!), but I’m very proud of our work.

Some fun things about Mir Ion:

- We can fully deserialize Ion at compile-time (via D’s CTFE functionality)

- We’re one of the fastest JSON parsing libraries (and one of the most memory efficient too — we actually store all JSON data in memory as Ion data, which is vastly more efficient)

- We’re nearly 100% compliant to all of the upstream test cases (our main issue is that we’re often too lax on spec, and allow files that are invalid through)

- The entire library is (nearly) all `@nogc`, thanks to the Mir standard library

If anyone has any questions on Mir Ion, feel free to shoot me a line at harrison (at) 0xcc.pw

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#195
post #100

Earlier quoted context omitted.

Yes it is. Message schemas are made by humans. Most of these messages will be extended in a backwards compatible manner over the life of a project rather than replaced entirely so their IDs don’t change. That’s kinda the point of protobufs and its successors. I’ve probably generated 100 IDs over my lifetime.

Which puts it on the same order of magnitude as the number of people on the planet. If every person alive generated a schema (or if 1/100th of all people generate 100 IDs each like you) then we'd have a small number of collisions. More likely you'd get large numbers of schema like that if there's a widespread application of a protocol compiler that generates new schema programmatically, e.g. to achieve domain separat…

It's only really a problem if you use the IDs in the same system. It's highly unlikely that you'd link 4B schemas into a single binary. And anyway, if you do have a conflict, you'll get a linker error.

Cap'n Proto type IDs are not really intended to be used in any sort of global database where you look up types by ID. Luckily no one really wants to do that anyway. In practice you always have a more restricted set of schemas you're interested in for your particular project.

(Plus if you actually created a global database, then you'd find out if there were any collisions...)

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#196
post #96

Earlier quoted context omitted.

> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

When you reach the 4 billionth version of your protocol?

All versions of the same protocol have the same ID. That is the point of IDs -- to link together different versions of the protocol.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#198
post #146
post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

Protobufs is a boring old tag-length-value format. It's kind of the worst of both worlds because it has no type information encoded in to it, meaning it's useless without the schema, while still having quite a bit of overhead. Capn'Proto is more like a formalization of C structs in that new fields are only added to the end. If memory serves, on the wire there is no tag, type or length info (for fixed size field types…

Mostly right. Allow me to provide some wonky details.

Protobuf uses tag-type-values, i.e. each field is encoded with a tag specifying the field number and some basic type info before the value. The type info is only just enough information to be able to skip the field if you don't recognize it, e.g. it specifies "integer" vs. "byte blob". Some types (such as byte blob) also have a length, some (integer) do not. Nested messages are usually encoded as byte blobs with a length, but there's an alternate encoding where they have a start tag and an end tag instead ("start group" and "end group" are two of the basic types). On one hand, having a length for nested messages seems better because it means you can skip the message during deserialization if you aren't interested in it. On the other hand, it means that during serialization, you have to compute the length of the sub-message before actually serializing it, meaning the whole tree has to be traversed twice, which kind of sucks, especially when the message tree is larger than the L1/L2 cache. Ironically, most Protobuf decoders don't actually support skipping parsing of nested messages so the length that was so expensive to compute ends up being largely unused. Yet, most decoders only support length-delimited nested messages and therefore that's what everyone has to produce. Whoops.

Now on to Cap'n Proto. In a given Cap'n Proto "struct", there is a data section and a pointer section. Primitive types (integers, booleans, etc.) go into the data section. This is the part that looks like a C struct -- fields are identified solely by their offset from the start of the data section. Since new fields can be added over time, if you're reading old data, you may find the data section is too small. So, any fields that are out-of-bounds must be assumed to have default values. Fields that have complex variable-width types, like strings or nested structs, go into the pointer section. Each pointer is 64 bits, but does not work like a native pointer. Half of the pointer specifies an _offset_ of the pointed-to object, relative to the location of the pointer. The other half contains... type information! The pointer encodes enough information for you to know the basic size and shape of the destination object -- just enough information to make a copy of it even if you don't know the schema. This turns out to be super-important in practice for proxy servers and such that need to pass messages through without necessarily knowing the details of the application schema.

In short, both formats actually contain type information on the wire! But, not a full schema -- only the minimal information needed to deal with version skew and make copying possible without data loss.

Post reply on HN