131–140 of 240 posts
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
Thank god.. JSON for config files without comments is so awful.
Earlier quoted context omitted.
That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…
Ouch that’s rough. One nice bit of IPv6 is that it doesn’t allow fragmentation. It often much nicer to get no message or an error than subtly missing data.
This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
But Douglas Crockford just don't want to innovate anything, just like Gruber didn't want to make a proper specification of the Markdown format.
Sometimes people are keeping innovation back. Fortunately this did not happend with html.
The main thing missing from the text format is a magic and version number. At least the binary format has it.
Earlier quoted context omitted.
If you generate randomly then, due to the birthday paradox, after generating sqrt(N) values you have a reasonable chance of collision. The birthday paradox is named after the non-intuitive fact that with just 32 people in a room you have > 50% of 2 people having a birthday on the same day of the year.
Does birthday paradox apply here? It’s about any pair of people having the same birthday, whereas in this case you need someone else with a specific birthday. For example, if you generate 2 numbers and they are the same, but are different to the capnproto number, that’s a collision but doesn’t actually matter. EDIT: It does apply, I misunderstood what the number was being used for.
Did anything ever become of the lispy language that was being built using Ion as its homoiconic syntax? I'm afraid I can't recall what it was called. Fusion maybe?
I wonder what's the performance relative to native JSON parsers?
We have done some work on performance comparisons with the ion-java-benchmark-cli tool (https://github.com/amzn/ion-java-benchmark-cli). Right now you can compare JSON serialized with Jackson and there is a pull request (https://github.com/amzn/ion-java-benchmark-cli/pull/27) for comparing against CBOR that should be merged soon.
We are always happy to hear suggestions for what is useful in this area.
Earlier quoted context omitted.
> It’s a 64bit random number so it’ll never have unintentional collisions. It'll have unintentional collisions if you ever generate more than 4 billion of these random numbers. That's not inconceivable.
Yes it is. Message schemas are made by humans. Most of these messages will be extended in a backwards compatible manner over the life of a project rather than replaced entirely so their IDs don’t change. That’s kinda the point of protobufs and its successors. I’ve probably generated 100 IDs over my lifetime.
Earlier quoted context omitted.
Does birthday paradox apply here? It’s about any pair of people having the same birthday, whereas in this case you need someone else with a specific birthday. For example, if you generate 2 numbers and they are the same, but are different to the capnproto number, that’s a collision but doesn’t actually matter. EDIT: It does apply, I misunderstood what the number was being used for.
It does apply, according to https://www.johndcook.com/blog/2017/01/10/probability-of-sec...