Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

amzn.github.io

Post reply on HN
Print view View original

211–220 of 240 posts

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#211

Post by biztos »

pdpi wrote: post #207
programd wrote: post #206

"Zero and negative dates are not valid, so the earliest instant in time that can be represented as a timestamp is Jan 01, 0001" That seems to be...a problem? How do you deal with archeological dates, of which there are many, in Ion?

That’s an interesting question. On the one hand, it feels weird that you can’t represent those dates at all. On the other hand, representability of a given date becomes progressively less useful the further back in time you go, and stuff becomes really gnarly once you go past the Julian calendar in 45BC. Also, simplifying to “no dates before Jan 1 0001” has very little impact on applications dealing with the modern-i…

Just to give one example, in Thailand right now it's the year 2564.

1 BC for some is not "-1" for everyone.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#212

Post by usrusr »

seanclayton wrote: post #185
kwertyoowiyop wrote: post #179

Earlier quoted context omitted.

The dominance of JSON shows that Crockford made some good decisions, even though we may not agree with them on any given day.

The dominance of JSON just shows that JS is dominant.

Even JS stopped parsing JSON as a subset JS a long time ago. JSON lineage has been irrelevant in terms of popularity transmission ever since people stopped doing var jsonobj = eval(jsonstring);

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#213

Post by wisty »

travisd wrote: post #19
wisty wrote: post #16

I scanned the docs, and can't see what happens if you alter your data schema. Anyone know?

Seems like you have to handle that yourself. The serialized data includes the type, so your app code might have to have logic a la “if type1: … else: …” after parsing it.

OK, so it's one of the more flexible ones (like those binary jsons) rather than something like protobuf. I guess that should have been obvious from "self-describing".

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#214

Post by indymike »

seanclayton wrote: post #185
kwertyoowiyop wrote: post #179

Earlier quoted context omitted.

The dominance of JSON shows that Crockford made some good decisions, even though we may not agree with them on any given day.

The dominance of JSON just shows that JS is dominant.

> The dominance of JSON just shows that JS is dominant.

I don't know that's the case... I've used JSON in lots of non-JS languages because it just works, and errors rarely are caused by mismatches in how JSON behaves in language X and language Y. A lot of that is that it is simple, and rigid.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#215

Post by agumonkey »

angstrom wrote: post #52
hliyan wrote: post #44

This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…

And to top it off you could fit the entire message into whatever the MTU of your network supported. Cap it at 1500 bytes and subtract the overhead for the frame headers and you get an extremely tight TCP/IP sequence stream that buffers through 16MB without needing to boil the ocean for a compound command sequence. Having been in industry only 2 decades it amuses me how many times this gets rediscovered.

Smells like engineering

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#216

Post by dang »

throwoutway wrote: post #187
dang wrote: post #74

Earlier quoted context omitted.

Thanks! Macroexpanded: Amazon Ion - https://news.ycombinator.com/item?id=23921610 - July 2020 (110 comments) Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset - https://news.ycombinator.com/item?id=11546098 - April 2016 (163 comments)

What do you use for the macroexpansion? There are a hundred odd tasks like this that I need to create macros for!

I mean that metaphorically but I do have a bunch of keyboard shortcuts (in a browser extension) that make finding these, and formatting the comments, much faster.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#217

Post by indymike »

nly wrote: post #152
sirk390 wrote: post #119

timestamps and decimal are the two most useful additions compared to json. They would be nice to add to json if that is somehow possible.

JSON numbers, just like all human readable formats, are decimal... it's not like binary double values are printed out in to JSON in hex or base64 Sure 99% of decoders convert them to and from binary doubles, but that's purely an implementation choice.

> JSON numbers, just like all human readable formats, are decimal...

All JSON numbers are implemented as integers or floating point, and as a result, have to be cast as a decimal (a decimal type is generally something that meets this specification: http://speleotrove.com/decimal/) when you import them.

Decimal types differ from floating point types in three ways: they are accurate, and they take into account rounding rules and precision. Decimal math is slower, can have greater precision and is better suited to domains where finite precision is needed. Floating point is faster, but is not as precise, so it's good for some scientific uses... or where perfect precision isn't important but speed is... say 3d graphics.

I've billed lots of hours over the years fixing code where a developer used floats where they should have used decimals. For example, if you are dealing with money, you probably want decimal. It's one of those problems like trying to parse email addresses with a regex or rolling your own crypto... it will kind a work until someone finds out it really doesn't (think accounting going, our numbers are off by random amounts, WTF?).

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#218

Post by indymike »

hirundo wrote: post #15

It seems like an odd choice to make the type "metadata" a prefix to the value, rather than a separate field. It feels like overloading. What's the advantage?

It tells you how to load the value and can be human readable for audit purposes. example: degrees::'celsius'::100

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#219

Post by danachow »

dan-robertson wrote: post #210
danachow wrote: post #94

Earlier quoted context omitted.

OT but keyboard latency can and often is far below 50ms, more like 1ms. It seems to be a common misconception that denouncing mandates increased lag.

Is this a number that came from an actual benchmark or from some marketing material from a keyboard maker? I ask this because [1] finds latency (measured from touching the key to the usb packet arriving) of 15ms with the fastest keyboard and around 50ms with others, though apparently some manufacturers have since improved. Or are you talking about midi keyboards where I guess latency is more noticeable to users? [1]…

From the countless review and small time YouTube channels that test these things regularly.

I think that post must be a few years out of date - and moreover by its own admission doesn’t even test hardly any “gaming” keyboards. There is a tremendous amount of competition in keyboards that has been building for the past 10 years.

Input latency is now a marketing thing like horsepower, and there are reasonably reputable [1] places and countless small time YouTube reviewers that test these things. It’s not like it is difficult to improve latency, and now that it is something that is competitively marketed it is delivered on.

[1] https://www.rtings.com/keyboard/tests/latency

Personally I think it’s a bit ridiculous. This fetishization with minimizing latency to now sub-ms levels doesn’t necessarily lead to better performance as many top level gamers do not use the lowest latency level keyboards. But that doesn’t change the fact that modern mainstream gaming keyboards can hit a latency far below 50ms.

Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format

#220

Post by nly »

indymike wrote: post #217
nly wrote: post #152

Earlier quoted context omitted.

JSON numbers, just like all human readable formats, are decimal... it's not like binary double values are printed out in to JSON in hex or base64 Sure 99% of decoders convert them to and from binary doubles, but that's purely an implementation choice.

> JSON numbers, just like all human readable formats, are decimal... All JSON numbers are implemented as integers or floating point, and as a result, have to be cast as a decimal (a decimal type is generally something that meets this specification: http://speleotrove.com/decimal/ ) when you import them. Decimal types differ from floating point types in three ways: they are accurate, and they take into account roundin…

A binary double can hold any decimal value to 15 digits of precision, so as a serialisation format it's a bit of a non-issue... you just need to convert to decimal and round appropriately before doing any arithmetic where it matters.

And you're confusing JSON the format with typical implementations. Open a JSON file and you see decimal digits. There is no limit to the number of the digits in the grammar. Parsing these digits and converting them to binary doubles, for example, is actually slower than parsing them as decimals, because you have to do the latter anyway to accomplish the former. Almost all JSON libraries convert to binary (e.g. doubles) because of their ubiquitous hardware and software support...but some libraries like RapidJSON expose raw numeric strings out of the parser if you want to plug in a decimal library

Post reply on HN