Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

Post reply on HN
Print view View original

71–80 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#71

Post by tjonker »

wyc wrote: post #2

This reminds me a lot of Avro: https://avro.apache.org/docs/current/ They both have self-describing schemas, support for binary values, JSON-interoperability, basic type systems (Ion seems to support a few more field types), field annotations, support for schema evolution, code generation not necessary, etc. I think Avro has the additional advantages of being production-tested in many different companies, a fully-JSO…

A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format.

It's a core feature of Ion that the text and binary representations are isomorphic. You can take any Ion binary document and pretty-print it as an Ion text document that is exactly equivalent. You can edit that document and send it into your application, which will be guaranteed to be able to read it. Or you can take your hand-authored text data and transcode it into binary, and know that any Ion application can handle it without any extra effort.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#72

Post by amyjess »

tjonker wrote: post #71
wyc wrote: post #2

This reminds me a lot of Avro: https://avro.apache.org/docs/current/ They both have self-describing schemas, support for binary values, JSON-interoperability, basic type systems (Ion seems to support a few more field types), field annotations, support for schema evolution, code generation not necessary, etc. I think Avro has the additional advantages of being production-tested in many different companies, a fully-JSO…

A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format. It's a core feature of Ion that the text and binary representations are isomorphic. You can tak…

> A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format.

Also, field order matters in Avro but not in JSON. That bit me pretty hard once... fortunately, I found out that Python's JSON library lets you read a JSON file into an OrderedDict instead of a plain dict, so I was able to get around it.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#73

Post by haberman »

jcrites wrote: post #68
haberman wrote: post #60

To think about the difference between serialization formats, here's an analogy I hope will help. Protocol Buffers (and I think Thrift, and maybe Avro) are sort of like C or C++: you declare your types ahead of time, and then you take some binary payload and "cast" it (parse it actually) into your predefined type. If those bytes weren't actually serialized as that type, you'll get garbage. On the plus side, the fact t…

That's a great analogy! However, I do think strongly typed vs. weakly typed has a role in thinking about this, just a different dimension than the one you're describing. Let's say we come across a JSON structure that looks like this: {"start": "2007-03-01"} Is that a timestamp? Maybe! Does it support a time within the day? Perhaps I can write "2007-03-01T13:00:00" in ISO 8601 format if we're lucky. Can I supply a tim…

Ion has more data types than JSON, it's true. Ion has a timestamp type and JSON does not, so you could say it's "richer" if you want, but that just means "it has more types."

However I don't think it's accurate to say that the typing of Ion is any "stronger." Both Ion and JSON are fully dynamically typed, which means that types are attached to every value on the wire. It's just that without an actual timestamp type in JSON, you have to encode timestamp data into a more generic type.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#75

Post by haberman »

dietrichepp wrote: post #70
haberman wrote: post #60

To think about the difference between serialization formats, here's an analogy I hope will help. Protocol Buffers (and I think Thrift, and maybe Avro) are sort of like C or C++: you declare your types ahead of time, and then you take some binary payload and "cast" it (parse it actually) into your predefined type. If those bytes weren't actually serialized as that type, you'll get garbage. On the plus side, the fact t…

That's a good description, but I'd say that we have a strongly weakly typed axis and a statically dynamically typed axis here. Or I might actually prefer to name the first axis poorly richly typed. poorly typed richly typed dynamic CSV, INI JSON YAML, Ion static Bencode, ASN.1 Protobuf What I mean by "richly typed" is that you would never read a timestamp off the wire and not know that it's a timestamp. By comparison…

Sure, I think your "poorly typed" vs. "richly typed" axis just refers to how many built-in types it has. It's true that CSV and INI only have one type (string). And it's true that when more types are built in, you have fewer cases where you have to just stuff your data into a specially-formatted string.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#76

Post by jcrites »

haberman wrote: post #73
jcrites wrote: post #68

Earlier quoted context omitted.

That's a great analogy! However, I do think strongly typed vs. weakly typed has a role in thinking about this, just a different dimension than the one you're describing. Let's say we come across a JSON structure that looks like this: {"start": "2007-03-01"} Is that a timestamp? Maybe! Does it support a time within the day? Perhaps I can write "2007-03-01T13:00:00" in ISO 8601 format if we're lucky. Can I supply a tim…

Ion has more data types than JSON, it's true. Ion has a timestamp type and JSON does not, so you could say it's "richer" if you want, but that just means "it has more types." However I don't think it's accurate to say that the typing of Ion is any "stronger." Both Ion and JSON are fully dynamically typed, which means that types are attached to every value on the wire. It's just that without an actual timestamp type i…

The notions of "strong" and "weak" typing have never been particularly well-defined, but I think my usage is in line with their usual meaning: https://en.wikipedia.org/wiki/Strong_and_weak_typing

> Some programming languages make it easy to use a value of one type as if it were a value of another type. This is sometimes described as "weak typing".

Strong typing makes it difficult to use a value of one type as if it were another. In PHP, you can compare the integer value 1 to the string value "1" and the equality test returns boolean true. Conflating integer 1 and string "1" is weak typing. A data format that expresses the concept of the timestamp 1999-12-31T23:14:33.079-08:00 using the same fundamental type as the string "Party like it's 1999!" is what I would call weakly typed.

Ion does not make it easy to use a string as if it were a timestamp or vice versa. It has types like arbitrary precision decimals, or binary blobs, that can't easily be represented in a strongly-typed way in JSON. You can certainly invent a representation, like specifying strings as ISO 8601 for timestamps, or an array of numbers for binary -- actually, wait, how about a base64-encoded string instead? Where there's choice there's ambiguity. These concepts of "type" live in the application layer in JSON, instead of in the data layer like they do in Ion.

Note as well that stronger is my term. The Ion documentation says "richly-typed". Certainly Ion does not include every type in the world. Perhaps a future serialization framework might capture "length" with a unit of "meters", or provide a currency type with unit "dollars", and if that existed I'd call it stronger-(ly?)-typed or more richly typed than Ion. In that case, the data layer would prevent you from accidentally converting "3 inches" to "3 centimeters" by accident, since those would be different types. That would be stronger typing than an example where you simply have the integer 3, and it's the application's job to track which integers represent inches, and which represent centimeters. So perhaps "strong" and "weak" are not the best terms, so much as "stronger" and "weaker".

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#78

Post by tjonker »

haberman wrote: post #73
jcrites wrote: post #68

Earlier quoted context omitted.

That's a great analogy! However, I do think strongly typed vs. weakly typed has a role in thinking about this, just a different dimension than the one you're describing. Let's say we come across a JSON structure that looks like this: {"start": "2007-03-01"} Is that a timestamp? Maybe! Does it support a time within the day? Perhaps I can write "2007-03-01T13:00:00" in ISO 8601 format if we're lucky. Can I supply a tim…

Ion has more data types than JSON, it's true. Ion has a timestamp type and JSON does not, so you could say it's "richer" if you want, but that just means "it has more types." However I don't think it's accurate to say that the typing of Ion is any "stronger." Both Ion and JSON are fully dynamically typed, which means that types are attached to every value on the wire. It's just that without an actual timestamp type i…

I generally agree, except the "type" of JSON numbers isn't well-defined with respect to precision and binary-vs-decimal floating point representation. An application that cares deeply about either aspect of numbers can't rely on JSON alone to ensure that the values are properly interpreted by all consumers.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#79

Post by [deleted] »

Cshelton wrote: post #67
wyc wrote: post #44

Earlier quoted context omitted.

I find that many financial technology companies opt to store currency as strings. The small overhead is typically well worth freedom from floating-point errors.

Correct. We store as strings then derive a number from the string for sorting purposes.

[deleted]

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#80

Post by userbinator »

_wmd wrote: post #66
userbinator wrote: post #47

Almost every time I see yet another structured data format I'm surprised at the number of people who haven't ever heard of ASN.1, despite it forming the basis of many protocols in widespread use.

Usual ASN.1 caveat: parsing its specifications requires money and a lot of time, implementing many of its encodings (e.g. unaligned PER) is a lifetime's work, and even the simpler ones thousands of eyes haven't managed to get right despite years of effort (see OpenSSL, NSPR, etc) ASN.1 also has a million baroque types (VideotexString, anyone?) where most people just need "string", "small int", "big int", etc. Some mo…

Usual ASN.1 caveat: parsing its specifications requires money and a lot of time, implementing many of its encodings (e.g. unaligned PER) is a lifetime's work

...unless you're Fabrice Bellard, who apparently wrote one just because it was one of the minor obstacles on the way to writing a full LTE base station:

http://www.bellard.org/ffasn1/

Post reply on HN