Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

51–60 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#51

Post by dastbe »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default.

as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the minimum amount of information required.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#52

Post by angstrom »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence.

Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#53

Post by syspec »

Surprised I have not heard of this before, I'd love something to come along and give JSON a kick in the pants.

I do think JSON is the defacto standard, and it really does get the job done, but for some more advances uses something like this could really shine.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#54

Post by seniorsassycat »

n8ta wrote: post #40

I recently implemented a similar (simpler) format https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned -…

ion text is a good contender for JSON, YAML, TOML usecases. It's also a good way to present the binary to humans.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#55

Post by petilon »

I like the fact that you can annotate objects as well, not just literals. So this is valid:

    animal: Tiger:: {
       gender: 'F',
       weight: 450
    }
This solves the inheritance problem, i.e., if you have multiple subclasses how do you know which type to deserialize as?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#56

Post by imiric »

n8ta wrote: post #40

I recently implemented a similar (simpler) format https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned -…

> what's the text format even for?

Configuration files?

Not sure if that's an intended use case, but being more flexible than JSON and stricter than YAML seems ideal for configuration.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#57

Post by hliyan »

angstrom wrote: post #52
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without success, until someone figured out that the new market addition had caused the market data message to exceed the Ethernet frame size for the first time. Problem was not at the application layer or the transport, it was data link layer fragmentation! Figuring that out felt like solving a murder mystery (I wasn't the one who figured it out though).

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#58

Post by seniorsassycat »

trinovantes wrote: post #24

I wonder what's the performance relative to native JSON parsers?

Parsing ion text should be similar to json, it has the same characteristics. All JSON is valid ion text so you can even parse JSON with an ION parser.

The binary parser is much faster. All fields are length-prefixed so a parser doesn't have to scan forward for the next syntax element.

The ion parsers (lexer? not sure the right vocab) I've worked with have a `JSON.parse` equivalent that returns a fully realized object, a Map, Array, Int, ect but they also have a streaming parser that yields value by value. You can skip over values you don't need, step over structs or into structs without creating a Map or Array. That can be much faster.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#60

Post by yakkityyak »

clhodapp wrote: post #38

It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.

You should look into https://cuelang.org
Post reply on HN