Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

Post reply on HN
Print view View original

61–70 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#61

Post by tjonker »

Ericson2314 wrote: post #26
alextgordon wrote: post #17

Earlier quoted context omitted.

The use of != there is very confusing but what they mean is stores a precision along each number, not that -0 != -0.0 e.g. in Python: >>> from decimal import Decimal as D >>> 2 * D("1.0") Decimal('2.0') >>> 2 * D("1.000") Decimal('2.000') >>> D("1.0") == D("1.000") True

That just means == is a "lossy" equivalence relation. I rather the precision be truely observable----every number is "infinite precision". Once can always include natural as extra field if one cares about empirical precision.

I'm having a bit of trouble parsing this, but Ion decimal values are not "infinite precision". Every decimal has a very specific, finite precision. It's a standard "coefficient and exponent" model, with no specification-enforced limit on either.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#62

Post by [deleted] »

nikolay wrote: post #50
leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

Okay, but they did a really poor job marketing it in this release. Plus, if it's used within Amazon, why it's Java-only so far?

[deleted]

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#63

Post by jonhohle »

conradev wrote: post #22
leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

Sounds a lot like Apple's property list format, which shares almost everything you listed in common, except for annotations and symbol tables. Its binary format was introduced in 2002! Edit: Property lists only support integers up to 128 bits in size and double-precision floating point numbers. On top of those, Ion also supports infinite precision decimals.

Like Property Lists the binary format is TLV encoded as well. Ion has a more compact binary representation for the same data and additional types and metadata. Also, IIRC, Plist types are limited to 32-bit lengths for all data types. The binary Ion representation has no such restriction (though in practice sizes are often limited by the language implementation).

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#64

Post by sound_of_basker »

haberman wrote: post #60

To think about the difference between serialization formats, here's an analogy I hope will help. Protocol Buffers (and I think Thrift, and maybe Avro) are sort of like C or C++: you declare your types ahead of time, and then you take some binary payload and "cast" it (parse it actually) into your predefined type. If those bytes weren't actually serialized as that type, you'll get garbage. On the plus side, the fact t…

!!!! My understanding went up several orders of magnitude! Thank you!!

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#65

Post by _wmd »

duskwuff wrote: post #51
vparikh wrote: post #30

Wasn't this solved already by the BSON specification - http://bsonspec.org ? Sure this allows you a definition of types, but this could easily be done using standard JSON meta data for each field. I find BSON simpler and more elegant.

BSON is awful. * It doesn't have "true" types in the sense that Ion does. It's basically just a binary serialization of JSON, with extra stuff. * Despite being a binary format, it's actually bulkier than JSON in most situations. * It removes any semblance of canonicity from many representations. A number, for instance, can potentially be represented by any of at least 3 types (double, int32, and int64). * It has sign…

Most of this comes from BSON also being the internal storage format for a database server. For example, at least the redundant string NULs make it possible to use C library functions without copying, the unpacked ints allow direct dereferencing, etc.

I've no clue about the trailing NUL on the record itself, perhaps a safety feature?

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#66

Post by _wmd »

userbinator wrote: post #47

Almost every time I see yet another structured data format I'm surprised at the number of people who haven't ever heard of ASN.1, despite it forming the basis of many protocols in widespread use.

Usual ASN.1 caveat: parsing its specifications requires money and a lot of time, implementing many of its encodings (e.g. unaligned PER) is a lifetime's work, and even the simpler ones thousands of eyes haven't managed to get right despite years of effort (see OpenSSL, NSPR, etc)

ASN.1 also has a million baroque types (VideotexString, anyone?) where most people just need "string", "small int", "big int", etc.

Some more on BER parsing hell here: https://mirage.io/blog/introducing-asn1

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#67

Post by Cshelton »

wyc wrote: post #44
kyllo wrote: post #25

Earlier quoted context omitted.

Real decimal type - invaluable when working with currency What does JavaScript do with this though, just cast it to a float?

I find that many financial technology companies opt to store currency as strings. The small overhead is typically well worth freedom from floating-point errors.

Correct. We store as strings then derive a number from the string for sorting purposes.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#68

Post by jcrites »

haberman wrote: post #60

To think about the difference between serialization formats, here's an analogy I hope will help. Protocol Buffers (and I think Thrift, and maybe Avro) are sort of like C or C++: you declare your types ahead of time, and then you take some binary payload and "cast" it (parse it actually) into your predefined type. If those bytes weren't actually serialized as that type, you'll get garbage. On the plus side, the fact t…

That's a great analogy! However, I do think strongly typed vs. weakly typed has a role in thinking about this, just a different dimension than the one you're describing. Let's say we come across a JSON structure that looks like this:

  {"start": "2007-03-01"}
Is that a timestamp? Maybe! Does it support a time within the day? Perhaps I can write "2007-03-01T13:00:00" in ISO 8601 format if we're lucky. Can I supply a time zone? Who knows for sure? It's weakly typed data. The actual specification of that type of that field lives in a layer on top of JSON, if it's even specified at all. It might be "specified" only in terms of what the applications that handle it can parse and generate. I could drop that value into Excel and treat it as all sorts of different things.

Ion by comparison has a specific data type for timestamps defined in the spec [1]. The timestamp has a canonical representation in both text and binary form. For this reason, I know that "2007-02-23T20:14:33.Z" and "2007-02-23T12:14:33.079-08:00" are valid Ion timestamp text values. In this instance I would describe Ion as strongly typed and JSON as weakly typed. Or, as the Ion documentation puts it, "richly typed".

To make an analogy, weakly typed is the Excel cell that can store whatever value you put in it, or the PHP integer 1 which is considered equal to "1" (loose equality). Strongly typed is the relational database row with a column described precisely by the table schema. Weakly typed is the CSV file; strongly typed is the Ion document.

[1] http://amznlabs.github.io/ion-docs/spec.html

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#69

Post by mattdeboard »

tantalor wrote: post #41
jonhohle wrote: post #31

Earlier quoted context omitted.

One significant advantage is you can opt-in to sharing schemas - without requiring all consumers to have your schema. Like a lot of Amazon's internal data formats, Ion designed to support backwards compatible schemas as well (that is, adding additional fields does not break existing consumers). It has isomorphic text and binary representations as part of the standard making debugging or optimized transport a config o…

> without requiring all consumers to have your schema Then how is the client supposed to handle the data? Guessing? > backwards compatible schemas > text and binary representations > type system > maps well to several languages Protos have all these. > S-Expressions Okay? Is that useful?

> Okay? Is that useful?

I bet it's super useful if you have sexprs in your data.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#70

Post by dietrichepp »

haberman wrote: post #60

To think about the difference between serialization formats, here's an analogy I hope will help. Protocol Buffers (and I think Thrift, and maybe Avro) are sort of like C or C++: you declare your types ahead of time, and then you take some binary payload and "cast" it (parse it actually) into your predefined type. If those bytes weren't actually serialized as that type, you'll get garbage. On the plus side, the fact t…

That's a good description, but I'd say that we have a strongly weakly typed axis and a statically dynamically typed axis here. Or I might actually prefer to name the first axis poorly richly typed.

            poorly typed  richly typed
    dynamic CSV, INI          JSON          YAML, Ion
    static        Bencode, ASN.1      Protobuf
What I mean by "richly typed" is that you would never read a timestamp off the wire and not know that it's a timestamp. By comparison, with CSV or INI files, you just have strings everywhere. Formats on the richly typed side have separate and explicit types for binary blobs and text, for example.
Post reply on HN