71–80 of 240 posts
Am I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
I still use it in my UDP game servers, with added packet id if message exceeds max datagram length and has to be split
Previous discussions: https://news.ycombinator.com/item?id=11546098 https://news.ycombinator.com/item?id=23921610
Amazon Ion - https://news.ycombinator.com/item?id=23921610 - July 2020 (110 comments)
Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset - https://news.ycombinator.com/item?id=11546098 - April 2016 (163 comments)
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
I had exactly the same implementation except that type / version belonged to the whole message and would map to appropriate binary buffer in memory. No real de/serialization was needed. I still use it in my UDP game servers, with added packet id if message exceeds max datagram length and has to be split
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?
Earlier quoted context omitted.
And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.
That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…
Earlier quoted context omitted.
I had exactly the same implementation except that type / version belonged to the whole message and would map to appropriate binary buffer in memory. No real de/serialization was needed. I still use it in my UDP game servers, with added packet id if message exceeds max datagram length and has to be split
The one concern I'd have with this format is a length field getting corrupted in transit and causing an out-of-bounds memory access. The network protocols' checksums won't save you 100% of the time, especially if there's bad hardware in the loop. If every field is fixed length this is less of a concern, of course; you might get bad data but you won't get e.g. a string with length 64M.