Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

231–240 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#231

Earlier quoted context omitted.

That's a list union[number, string, object] or list[Any], not a sum type, no? This `data X = A | B [A, B, ...]` Is a list containing a sum type: list[X]

There is no such thing in JSON or Ions as defining this "X" schema somewhere. So I may as well say that your [A,B,...] is a list[Any]. Now, I wouldn't actually call it a list of any, I would say you proved my point for me. Your example is functionally the same as mine. I would give this example: `[A, B, ...]` and say that that is a list of sum types. You may say "no no no! Only now is it a list of sum types!": `data…

> But my point is that there is no JSON/Ion equivalent of your `data X = A | B

No one disagrees - it's just that we complain about this. We _want_ to have such an equivalent.

> Saying that JSON/Ions don't support sumtypes is like saying JSON doesn't support "NonNegativeInteger" type.

Correct. But your conclusion is wrong. You seem to assume that no one has a problem with the fact that JSON doesn't support a "NonNegativeInteger" type. But I at least would happily use a format that explicitly supports that.

I mean, look at ION. Json doesn't support the concept of (restricted) integers, but ION extends JSON and offers this type. That's great, because it means if a library reads an integer field, it can map it to an integer and knows that there are constraints.

This is a _very_ relevant issue. Many json libraries in the past have had bugs or could be ddos-ed by feeding them json with large numbers, since the json spec does not constrain the size of numbers.

In that sense, ION could have _also_ added support for "NonNegativeInteger" or sumtypes, or other specific types, but they haven't. And since sumtypes are very fundamental, we complain about it more than we would complain about the lack of "NonNegativeInteger".

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#232
I don't like ion. The added features (over json) don't pull their weight. Symbols, annotations and the binary format all add significant complexity but don't make the format much better. As a consequence of the added complexity language support is poor.

For RPC the binary encoding compares poorly to external schema formats like protobuf. In this context binary ion is a poorly compressed text format.

I don't think the partial document read capability of the binary format is all that important, but I've never worked on an application that would benefit from it either.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#233

Earlier quoted context omitted.

But decimal are way more useful as they can represent currency amounts. It would be strange to show a currency amount like "3/4" or "11/12". Personally, the two datatypes I have always been adding manually to json are datetimes and decimals (from python)

A currency amount is just a rational number with "1000000" as a denominator. This is the correct representation, and how Google or the blockchain do it.

I don't thing there is a more "correct" representation. Representing it as a string is equaly correct. Blockchain or banking often represents money as an integer with the smallest divisible unit (cent, or satoshi), but it is not applicable here because there is no smallest divisible unit.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#235
post #152

timestamps and decimal are the two most useful additions compared to json. They would be nice to add to json if that is somehow possible.

JSON numbers, just like all human readable formats, are decimal... it's not like binary double values are printed out in to JSON in hex or base64 Sure 99% of decoders convert them to and from binary doubles, but that's purely an implementation choice.

If you only look at the json format it's true. But it would be way worse if some decoders returned decimals and some returned floats.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#236

Earlier quoted context omitted.

The link I posted was 2017. The site you link gives quite different ratings. I assume partly it is different methodology (the site you link tries to account for key travel somehow and they do something with a display and try to account for display latency rather than using a logic analyzer), but I’m not really sure. For some keyboards in common: - apple magic keyboard (? vs 2017) 15ms vs 27ms - das keyboard (3 vs S p…

I know it is 2017, but that is a very long time in the gaming/mech keyboard market. I remember just about 10 years ago when mech keyboards were a niche for weirdos and a few others that swore by their Model M's - you now can buy these in Walmart. The point about discrepancy is well taken* but I think the bigger point is on the rtings list the number of offerings that are an order of magnitude lower latency - such tha…

Why is using a high speed camera and a logic analyzer less viable than measuring the end to end latency and trying to subtract the computer part of it? Or are you suggesting that a solenoid should be used to press the key instead of a finger?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#237
post #57

Earlier quoted context omitted.

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without s…

How did you solve the problem?

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#238
post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

I assume you were using C++? I'm not sure what you describe is possible these days due to UB. At the very least just casting bytes received over the wire to a type is UB, so you technically need a memcpy() and hope that the compiler optimises it out.

It is not UB if it is correctly aligned.

It is UB if the underlying dynamic type is not compatible from the access type

As the bytes are coming from the kernel, you get to decide the dynamic type.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#239

Earlier quoted context omitted.

I know it is 2017, but that is a very long time in the gaming/mech keyboard market. I remember just about 10 years ago when mech keyboards were a niche for weirdos and a few others that swore by their Model M's - you now can buy these in Walmart. The point about discrepancy is well taken* but I think the bigger point is on the rtings list the number of offerings that are an order of magnitude lower latency - such tha…

Why is using a high speed camera and a logic analyzer less viable than measuring the end to end latency and trying to subtract the computer part of it? Or are you suggesting that a solenoid should be used to press the key instead of a finger?

The latter. Even the linked article points out the inherent limitations.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#240

Earlier quoted context omitted.

I know the differences between untagged and tagged unions, I'm trying to provide a minimal example without distracting details but sure we can talk about tagged unions. Here is a list of tagged unions, so I once again point out that sum types are "supported" in JSON/ions just as much as any other data type: [ {tag: "a", foo: 1}, {tag: "b", bar: "hi", baz: 2}, {tag: "a", foo: 3}, {tag: "a", foo: 4}, {tag: "a", foo: 5}…

By that reasoning, why have structs? You can just make an array of size-2 arrays that give the key and the value.

Okay I'll admit, that's a good point.
Post reply on HN