Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

Finally! I've had to live the JSON nightmare since I left Amazon.

Some of the benefits over JSON:

* Real date type

* Real binary type - no need to base64 encode

* Real decimal type - invaluable when working with currency

* Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo').

* Text and binary format

* Symbol tables - this is like automated jsonpack.

* It's self-describing - meaning, unlike Avro, you don't need the schema ahead of time to read or write the data.

That's not true. Avro at least existed at the time. However they wanted something self-describing to replace JSON/XML usage. Avro is better suited as a data storage format rather than a transit oriented format. Of course, both Ion and Avro can be used for either, but Avro will give you better compression on disk, but Ion is less cumbersome since it doesn't require a schema

Ion's advantage is that it's both strongly-typed with a rich type system, as well as self-describing.

Data formats like JSON and XML can be somewhat self-describing, but they aren't always completely. Both tend to need to embed more complex data types as either strings with implied formats, or nested structures. (Consider: How would you represent a timestamp in JSON such that an application could unambiguously read it? An arbitrary-precision decimal? A byte array?) I'm not familiar with EDN, but it appears to be in a similar position as JSON in this regard. ProtocolBuffers, Thrift, and Avro require a schema to be defined in advance, and only work with schema-described data as serialization layers. Ion is designed to work with self-describing data that might be fairly complex, and have no compiled-ahead-of-time schema.

Ion makes it easy to pass data around with high fidelity even if intermediate systems through which the data passes understand only part of the data but not all of it. A classic weakness of traditional RPC systems is that, during an upgrade where an existing structure gains an additional field, that structure might pass through an application that doesn't know about the field yet. Thus when the structure gets deserialized and serialized again, the field is missing. The Ion structure by comparison can be passed from the wire to the application and back without that kind of loss. (Some serialization-based frameworks have solutions to this problem too.)

One downside is that its performance tends to be worse than schema-based serialization frameworks like Thrift/ProtoBuf/Avro where the payload is generally known in advance, and code can be generated that will read and deserialize it. Another downside is that it's difficult to isolate Ion-aware code from the more general purpose "business logic" in an application, due to the absence of a serialization layer producing/consuming POJOs; instead it's common to read an Ion structure from the wire and access it directly from application logic.

The use of != there is very confusing but what they mean is stores a precision along each number, not that -0 != -0.0

e.g. in Python:

    >>> from decimal import Decimal as D
    >>> 2 * D("1.0")
    Decimal('2.0')

    >>> 2 * D("1.000")
    Decimal('2.000')

    >>> D("1.0") == D("1.000")
    True