Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#161

Post by serge2k »

tveita wrote: post #143
jonhohle wrote: post #42

Big congrats to Todd, Almann, Chris, Henry, and everyone else who made this happen. Several years ago, I wouldn't have imagined this possible and I'm a little bummed that I left before it happened. Like leef said above, I'm glad to have Ion as an option again.

I am curious why you "wouldn't have imagined this possible", is the reason technical or political?

Political.

Amazon doesn't open source things, as a general rule. It can be done but it is a lot of jumping through hoops and they generally need good reasons to do it (as opposed to a lack of good reasons not to).

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#162

Post by School-Cotton »

serge2k wrote: post #160
sswezey wrote: post #137

Earlier quoted context omitted.

I doubt it, when I was there Ion was only used by only a handful of Java teams doing backend work. It was also horribly documented and supported at the time (3.5 years ago).

I left < a year ago and never heard of ion.

What did you work on?

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#163

Post by dgreensp »

efaref wrote: post #130
leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

You could have used CBOR for many of those things ( http://cbor.io/ ).

Thanks for the link!

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#164

Post by rurban »

duskwuff wrote: post #51
vparikh wrote: post #30

Wasn't this solved already by the BSON specification - http://bsonspec.org ? Sure this allows you a definition of types, but this could easily be done using standard JSON meta data for each field. I find BSON simpler and more elegant.

BSON is awful. * It doesn't have "true" types in the sense that Ion does. It's basically just a binary serialization of JSON, with extra stuff. * Despite being a binary format, it's actually bulkier than JSON in most situations. * It removes any semblance of canonicity from many representations. A number, for instance, can potentially be represented by any of at least 3 types (double, int32, and int64). * It has sign…

Yes, and not only that. It also inherently insecure, while JSON is together with msgpack the only fast and secure serialization format out there. The problem is the encoding of objects and code without any checksumming, so it can be trivially tampered with, leading to very nice exploits, mostly remotely.

See e.g. https://metacpan.org/pod/Cpanel::JSON::XS#SECURITY-CONSIDERA... I need to add ion to this security matrix.

YAML does most of those and does more and can be made quite secure by limiting the allowed types to the absolute and trusted minimum, but this e.g. not implemented in the perl, only the python backend. By default YAML is extremely insecure.

There are more new readable and typed JSON variants out there. E.g. jzon-c should be faster than ion, but there are also Hjson and SJSON. See https://github.com/KarlZylinski/jzon-c

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#165

Post by rurban »

userbinator wrote: post #47

Almost every time I see yet another structured data format I'm surprised at the number of people who haven't ever heard of ASN.1, despite it forming the basis of many protocols in widespread use.

We heard of it and we despise it. It's the most horrible structured data format out there in the wild. Even worse than XML, and this is quite something.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#166

Post by tjonker »

haberman wrote: post #81
tjonker wrote: post #78

Earlier quoted context omitted.

I generally agree, except the "type" of JSON numbers isn't well-defined with respect to precision and binary-vs-decimal floating point representation. An application that cares deeply about either aspect of numbers can't rely on JSON alone to ensure that the values are properly interpreted by all consumers.

That is a good point in that it is a very accurate reading of the JSON spec. In practice many (even most) JSON implementations don't give applications access to any precision beyond what an IEEE double can represent. So while you may take advantage of arbitrary precision in JSON and be fine according to the spec, your users will probably suffer data loss unless they are very picky about what JSON library they use. Fo…

It's more than just precision, it's making sure that the same value comes out that went in, and that things haven't been subtly altered via unintended conversions between decimal and binary floating-point representations. Obviously this is quite important when you've got both text and binary formats.

Some applications really need decimal values, and some really need IEEE floats. Ion can accurately and precisely denote both types of data, making it easier to ensure that the data is handled properly by both reader and writer.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#167

Post by tjonker »

jasonpeacock wrote: post #140
kazinator wrote: post #84

I Consider this Harmful (TM) and will oppose the adoption in every organization where I have an opportunity to voice such. (In its present form, to be clear!) There is no need to have a null which is fragmented into null.timestamp, null.string and whatever. It will complicate processing. Just because you know the type of some element is timestamp, you must worry whether or not it is null and what that means. There sh…

Actually, ION was originally developed for C++ and took quite a while to gain Java support. Source: I was there.

Sorry, this isn't true. The Java and C++ implementations were developed simultaneously, by different authors. The Java side has always been more full-featured, and has always had a larger user base by at least an order of magnitude. Today IonJava is among the most widely-consumed libraries within Amazon.

Source: I am one of the primary authors.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#170

Post by swaranga »

sswezey wrote: post #137
School-Cotton wrote: post #111

Earlier quoted context omitted.

I'm sure there are ion bindings for every language in common use at Amazon. But a huge percentage of Amazon code is Java, so presumably this one was the best maintained and documented.

I doubt it, when I was there Ion was only used by only a handful of Java teams doing backend work. It was also horribly documented and supported at the time (3.5 years ago).

I am still in Amazon and Ion is definitely the most widely used library around. It has among the best documented code and some of the extensions that have been built on top of Ion are simply amazing.
Post reply on HN