Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

151–160 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#151

Post by [deleted] »

garmaine wrote: post #93
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.

[deleted]

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#152

Post by nly »

sirk390 wrote: post #119

timestamps and decimal are the two most useful additions compared to json. They would be nice to add to json if that is somehow possible.

JSON numbers, just like all human readable formats, are decimal... it's not like binary double values are printed out in to JSON in hex or base64

Sure 99% of decoders convert them to and from binary doubles, but that's purely an implementation choice.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#153

Post by tlocke »

One problem with Ion is that it doesn't have a map type, but instead a struct type that allows duplicate keys. I created Zish https://github.com/tlocke/zish as a serialization format that addresses the shortcomings of JSON and Ion. Any comments / criticisms welcome.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#154

Post by pcarolan »

dboreham wrote: post #77
armchairhacker wrote: post #68

Earlier quoted context omitted.

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

This is protocol buffers + a global type registry. I worked on such a system.

Is it public? Id love to learn more about it.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#155

Post by mtrovo »

armchairhacker wrote: post #68
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…

I think the system OP is describing is a little bit more complex. You're not just describing message types, you also have message templates; a template declares a message type and a set of prefilled fields. You save data by just sending the subset of fields that are actually changing, which is a very good abstraction for market data. The template is hydrated on the protocol parsing layer so your code only has to deal with message types itself.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#156

Post by mtrovo »

nly wrote: post #148
ktzar wrote: post #127

Earlier quoted context omitted.

Is it FIX messages? https://en.wikipedia.org/wiki/Financial_Information_eXchange It's a good idea, extensible (ranges available for banks to implement their own codes), and fast.

Old school texty FIX is incredibly slow. FAST FIX is faster but not fun to use. Largely SBE has won adoption on the market data side, with huge platforms like Euronext (biggest on Europe) using it.

I stopped working in the area on the age of FAST FIX, which was extremely good for the time. Do you know what are the differences to SBE?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#157

Post by andylynch »

jkhdigital wrote: post #79
hliyan wrote: post #57

Earlier quoted context omitted.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…

Classic example of a leaky abstraction, and the principle that implementation details inevitably become undocumented API behavior.

I’ve worked on systems like this too - the max packet size is very well documented. Then post trade it all gets turned into FIXML which somehow manages to be both more verbose and less readable.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#158

Post by mtrovo »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Don't know if you're describing the original FIX itself with the TCP connection. On FAST FIX they got rid of the TCP connection and market data was sent over UDP using several parallel connections, data was reordered on the client side at consumption time and it only used a TCP connection to recover data when a sequence gap was found.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#159

Post by [deleted] »

boxfire wrote: post #92
NavinF wrote: post #90

Earlier quoted context omitted.

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically

[deleted]

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#160

Post by hliyan »

mtrovo wrote: post #158
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

Don't know if you're describing the original FIX itself with the TCP connection. On FAST FIX they got rid of the TCP connection and market data was sent over UDP using several parallel connections, data was reordered on the client side at consumption time and it only used a TCP connection to recover data when a sequence gap was found.

Actually, even FAST was too slow for us. This was a proprietary messaging middleware library. And this particular market data feed was the direct one into the matching engine itself. For the rest of the system, we used a sort of reliable multicast using UDP for the first transmission and TCP for missed messages. We initially tried out a Gossip/Epidemic protocol but that didn't work out too well.
Post reply on HN