The latest format for Kindle eBooks, KFX, is based on this.
Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
101–110 of 240 posts
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#102Nice!
This thing is actually sane and thought through. A first for serialization formats. They're usually a shitshow.
(Should have gone with 'rational' instead of 'decimal', though. Decimal will be too painful to implement accross languages and implementations. Java bias?)
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#103I recently implemented a similar (simpler) format https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned -…
Ion will be even better for (structured) logging if this proposal for templates ever happens. https://github.com/amzn/ion-docs/pull/104 Looks like no one’s even so much as commented on it in the last year, so it might have been abandoned.
Disclosure: I manage the Ion and PartiQL teams at Amazon.
This proposal hasn’t been abandoned. We hope to post an update soon!
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#104Am I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/
I like base64 because it’s the de-facto standard, and data size (in places where I’d use base64) isn’t a main concern for me.
Yeah I get tired of reinvention of everything for tiny gains in size/performance.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#105It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.
Others have mentioned Protobuf and Capnproto's support. Avro has them too, they're called Union.
It seems that sum types are the norm, actually.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#106I like the fact that you can annotate objects as well, not just literals. So this is valid: animal: Tiger:: { gender: 'F', weight: 450 } This solves the inheritance problem, i.e., if you have multiple subclasses how do you know which type to deserialize as?
I believe that this is exactly how Jackson serialized Ion handles subtype polymorphism.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#107Earlier quoted context omitted.
You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)
> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.
>It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers.
If it's 64 bit, doesn't that mean you'd need to generate ~10000000000000000000000000000000000000000000000000000000000000000 (2^64) of those numbers to have a collision, not 2^32?
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#108It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.
Others have mentioned Protobuf and Capnproto's support. Avro has them too, they're called Union. It seems that sum types are the norm, actually.
Those do now but I believe that all of them added support years after their initial versions
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#109Earlier quoted context omitted.
> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.
>It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. If it's 64 bit, doesn't that mean you'd need to generate ~10000000000000000000000000000000000000000000000000000000000000000 (2^64) of those numbers to have a collision, not 2^32?
If you generate randomly then, due to the birthday paradox, after generating sqrt(N) values you have a reasonable chance of collision.
The birthday paradox is named after the non-intuitive fact that with just 32 people in a room you have > 50% of 2 people having a birthday on the same day of the year.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#110Earlier quoted context omitted.
Doesn't it trivially have "sum types" since it's just arbitrary self-describing data? i.e. nobody is stopping you from passing around objects in such a way: {a:1} {a:{b:2}} {a:4} {a:{b:4}} There's no static type layer over top of this, so it's inherently up to interpretation and whatever type system you want to use to describe this data, to be able to express that the values of `a` can be of type `number | {b: number…
> There's no static type layer over top of this Yeah, that's the problem. I mean, hey, why json? We could just use unstructured plaintext for everything and now we are free to do everything. But obviously that has its own drawbacks. Having built-in support for sumtypes means better and more ergonomic support from libraries, it means there is one standard and not different ways to encode things and it also means bette…
The point is that there's no reason to single out sumtypes here. Insofar as ions/json has support for arrays/objects/strings/numbers, it has exactly the same support for sumtypes, as in the example I showed above. Here is a list of "sumtype" `string | number | object`:
[{}, "hi", 1, 2, 3, "yo", {a: "bc"}]