Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

121–130 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#121

Post by the_girabbit »

valenterry wrote: post #114
the_girabbit wrote: post #46

Earlier quoted context omitted.

Genuine question—why would you need a sum type in a self-describing data format?

Well, there are already sumtypes, just only specific builtin ones, not custom ones. E.g. booleans are sumtypes (true | false). Everything else that is nullable is also a sumtype (e.g. number | null). I think it should be pretty obvious how these are helpful and why they are needed no?

Yeah, but it’s a schema-less, self-describing data format. It’s not like a specific position in a data stream has a requirement to be a specific type.

I can see why sum types would be useful in a schema or for the elements of a collection that is required to be homogeneous (ie. List).

For what use case would one use custom sum types in a schema-less data format?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#122

Post by seniorsassycat »

ComputerGuru wrote: post #42
seniorsassycat wrote: post #22

Earlier quoted context omitted.

Two representations of the same data structures. Ion text is like JSON, in fact all JSON is valid ion text. Ion text has comments, trailing commas, dates, and unquoted keys. It's a really good alternative to JSON, YAML, or TOML. Ion binary is compact and fast to parse. Values are length prefixed so the parser can skip over unneeded fields or structs, saving time parsing and memory allocated. Common string values, lik…

Do comments persist in binary serialization or is that a lossy one-way operation?

I think the ion java library includes a AST parser that includes comments, but the ION data model doesn't. The binary format cannot include comments.

I think many text parsers are missing libraries that edit documents in place, preserving formatting and comments.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#123

Post by seniorsassycat »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

ion schema is a type system that can validate ion values and it supports sum types.

https://amzn.github.io/ion-schema/docs/spec.html#union

The ion data model doesn't describe a schema or type system. It's a data structure where values are of a known type. In the binary format values are preceded by a type id, in the text format the syntax declares the type - "" for string, {} for struct. The data model doesn't declare what types a value could have, only the type it does have.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#124

Post by porker »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Fab story, thank you! I understood up to "Messages were exchanged over raw TCP. They contained their own application layer sequence number so that streams could resume after disconnection." Can you go into more details about how the sequence number and resuming after disconnection worked?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#126

Post by ratorx »

tomerv wrote: post #109
logicchains wrote: post #107

Earlier quoted context omitted.

>It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. If it's 64 bit, doesn't that mean you'd need to generate ~10000000000000000000000000000000000000000000000000000000000000000 (2^64) of those numbers to have a collision, not 2^32?

If you generate randomly then, due to the birthday paradox, after generating sqrt(N) values you have a reasonable chance of collision. The birthday paradox is named after the non-intuitive fact that with just 32 people in a room you have > 50% of 2 people having a birthday on the same day of the year.

Does birthday paradox apply here? It’s about any pair of people having the same birthday, whereas in this case you need someone else with a specific birthday.

For example, if you generate 2 numbers and they are the same, but are different to the capnproto number, that’s a collision but doesn’t actually matter.

EDIT: It does apply, I misunderstood what the number was being used for.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#127

Post by ktzar »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Is it FIX messages? https://en.wikipedia.org/wiki/Financial_Information_eXchange It's a good idea, extensible (ranges available for banks to implement their own codes), and fast.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#128

Post by garmaine »

Eelongate wrote: post #37

Did anything ever become of the lispy language that was being built using Ion as its homoiconic syntax? I'm afraid I can't recall what it was called. Fusion maybe?

I hope you get an answer, because this sounds very intriguing but google is failing me in finding any references to it.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#129

Post by elcritch »

hliyan wrote: post #57
angstrom wrote: post #52

Earlier quoted context omitted.

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…

Ouch that’s rough. One nice bit of IPv6 is that it doesn’t allow fragmentation. It often much nicer to get no message or an error than subtly missing data.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#130

Post by clhodapp »

spenczar5 wrote: post #117
clhodapp wrote: post #108

Earlier quoted context omitted.

Those do now but I believe that all of them added support years after their initial versions

I think you're incorrect: Avro had unions in version 1.0 [0], which is from 2012. Capnproto had unions back in 2013 [1]. That's from the v0.1 days, or maybe even earlier. Protobuf has had oneof support for about 7 years. They were added in version 2.6.0, from 2014-08-15 [2]. That's still 6 years after the initial public release in 2008, though, so this is maybe what you were thinking of? I don't know too many people…

Thanks for the references, friend!

And yes, I definitely am primarily thinking of protobuf, as I struggled with this back with version 2.5. I had the (apparently mistakenly) impression that Avro and Cap'n Proto (which I think actually first came out in this timeframe) were about on par.

Post reply on HN