Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

201–210 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#201

Post by o_bender »

hliyan wrote: post #165
ktzar wrote: post #127

Earlier quoted context omitted.

Is it FIX messages? https://en.wikipedia.org/wiki/Financial_Information_eXchange It's a good idea, extensible (ranges available for banks to implement their own codes), and fast.

It was a proprietary messaging middleware library. We actually found even FAST FIX slow.

FAST FIX protocol is terrible performance-wise, its format requires multiple branching at every field parsing. Even "high-performance" libraries like mFAST are slow: I recently helped a client to optimize parsing for several messages and got 8x speed improvement over mFAST (which is a big deal in HFT space).

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#202

Post by joshlemer »

ImprobableTruth wrote: post #174
joshlemer wrote: post #113

Earlier quoted context omitted.

What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list `

Sum types =/= union types. Sum types are also called 'tagged' or 'discriminable' unions because they have some way to discriminate between them. That is, if you have an element a of type A, a is not part of the sum type A + B because it's missing a tag. [5,"hello",3] has the type list (int ∪ string), not list (int + string). You can emulate the latter by manually adding a tag, but native support is much preferable.

I know the differences between untagged and tagged unions, I'm trying to provide a minimal example without distracting details but sure we can talk about tagged unions. Here is a list of tagged unions, so I once again point out that sum types are "supported" in JSON/ions just as much as any other data type:

    [
      {tag: "a", foo: 1},
      {tag: "b", bar: "hi", baz: 2},
      {tag: "a", foo: 3},
      {tag: "a", foo: 4},
      {tag: "a", foo: 5},
      {tag: "b", bar: "yo", baz: 6}
    ]

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#203

Post by joshlemer »

dunefox wrote: post #150
joshlemer wrote: post #113

Earlier quoted context omitted.

What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list `

That's a list union[number, string, object] or list[Any], not a sum type, no? This `data X = A | B [A, B, ...]` Is a list containing a sum type: list[X]

There is no such thing in JSON or Ions as defining this "X" schema somewhere. So I may as well say that your [A,B,...] is a list[Any].

Now, I wouldn't actually call it a list of any, I would say you proved my point for me. Your example is functionally the same as mine. I would give this example:

`[A, B, ...]`

and say that that is a list of sum types. You may say "no no no! Only now is it a list of sum types!":

`data X = A | B

[A, B, ...]`

But my point is that there is no JSON/Ion equivalent of your `data X = A | B`. Everyone in this comment tree is confusing the data itself with out-of-band schema over that data. "Sumtype" is nothing more than a fiction, or a schema. Saying that JSON/Ions don't support sumtypes is like saying JSON doesn't support "NonNegativeInteger" type. Sure it does! Here are some: 1, 2, 3, 10. What tooling or type system you use outside of the data itself to enforce constraints on the data types is orthogonal to the data format itself.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#204

Post by atlgator »

hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

We did the same in high fidelity flight simulators for a lot less money I'm sure.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#207

Post by pdpi »

programd wrote: post #206

"Zero and negative dates are not valid, so the earliest instant in time that can be represented as a timestamp is Jan 01, 0001" That seems to be...a problem? How do you deal with archeological dates, of which there are many, in Ion?

That’s an interesting question. On the one hand, it feels weird that you can’t represent those dates at all.

On the other hand, representability of a given date becomes progressively less useful the further back in time you go, and stuff becomes really gnarly once you go past the Julian calendar in 45BC.

Also, simplifying to “no dates before Jan 1 0001” has very little impact on applications dealing with the modern-ish world (with “modern” generously defined as “anything after the collapse of the Roman Empire”), and I can only assume applications dealing with earlier times could do with a more specialised representation for dates anyway.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#208

Post by Zamicol »

ralusek wrote: post #67
Zamicol wrote: post #59

Am I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/

I understand the case for Base91, but why hex over Base64? Base64 for readability and sticking to multiples of two, Base91 for maximum efficiency with readable ASCII.

Base 64 is good at nothing and bad at some things.

- Hex is human readable, case insensitive, not that "inefficient", and always aligns to bytes.

- Base 85 and basE91 are efficient.

- Bitcoin uses Base58 because they thought base 64 was too human unreadable. Ethereum uses Hex.

- Base 256 (bytes) is efficient and the native language of computers.

Base 64 is not efficient, not human readable, and not easy to encode.

The biggest problem with base 64 is that base 64 is not base 64. Are you doing base 64 with padding? Are you doing base 64 with URL safe characters or URL unsafe characters? Are you following the standard RFC 4648 bucket encoding, or are you using iterative divide by radix? I think a great place where the cracks show is JOSE, where for things like thumbprints there's a ton of conversion steps (UTF-8 key -> base 64 -> ASCII bytes -> digest (bytes) -> base 64 thumbprint).

My personal advise for 90% of projects considering looking at base 64 should just use Hex or bytes. If needing human readability, use Hex. Otherwise use binary.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#209

Post by elteto »

programd wrote: post #206

"Zero and negative dates are not valid, so the earliest instant in time that can be represented as a timestamp is Jan 01, 0001" That seems to be...a problem? How do you deal with archeological dates, of which there are many, in Ion?

What modern tech service (of the kind that would have use for Ion) is dealing with archaeological dates _at scale_? Honest question.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#210

Post by dan-robertson »

danachow wrote: post #94
Aeolun wrote: post #69

Earlier quoted context omitted.

> In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. If the order submission process depends on the manual press on the enter key (+/- 50ms) is there any point to that though?

OT but keyboard latency can and often is far below 50ms, more like 1ms. It seems to be a common misconception that denouncing mandates increased lag.

Is this a number that came from an actual benchmark or from some marketing material from a keyboard maker? I ask this because [1] finds latency (measured from touching the key to the usb packet arriving) of 15ms with the fastest keyboard and around 50ms with others, though apparently some manufacturers have since improved. Or are you talking about midi keyboards where I guess latency is more noticeable to users?

[1] https://danluu.com/keyboard-latency/

Post reply on HN