Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

Post reply on HN
Print view View original

41–50 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#41

Post by tantalor »

jonhohle wrote: post #31
tn13 wrote: post #16

This appears to be something in between of JSON and Protocol buffers. I wonder under what conditions Ion makes more sense than either of the JSON/PBuff.

One significant advantage is you can opt-in to sharing schemas - without requiring all consumers to have your schema. Like a lot of Amazon's internal data formats, Ion designed to support backwards compatible schemas as well (that is, adding additional fields does not break existing consumers). It has isomorphic text and binary representations as part of the standard making debugging or optimized transport a config o…

> without requiring all consumers to have your schema

Then how is the client supposed to handle the data? Guessing?

> backwards compatible schemas

> text and binary representations

> type system

> maps well to several languages

Protos have all these.

> S-Expressions

Okay? Is that useful?

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#42

Post by jonhohle »

Big congrats to Todd, Almann, Chris, Henry, and everyone else who made this happen.

Several years ago, I wouldn't have imagined this possible and I'm a little bummed that I left before it happened.

Like leef said above, I'm glad to have Ion as an option again.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#43

Post by leef »

wyc wrote: post #23
jcrites wrote: post #18

Earlier quoted context omitted.

What do you mean by they both have self-describing schemas? In order to read or write Avro data, an application needs to possess a schema for that data -- the specific schema that the data was written with, and (when writing) the same schema that a later reader expects to find. This means the data is not self-describing. Ion is designed to be self-describing, meaning that no schema is necessary to deserialize and int…

I think it depends on what level you're referring to. If you mean record-level, then I concede that it's not self-describing. However, looking at the suggested use cases, it seems that it's "self-describing" in that you'll always be able to decode data stored according to what the documentation recommends: "Avro data is always serialized with its schema. Files that store Avro data should always also include the schem…

The intent of the stored schema isn't really for self-description. A typical use case for Avro is data storage over long periods of time. It is expected that the schema will evolve at some point during this time. Therefore you still need to specify a target schema to read the data into which is allowed to be different than the stored schema. Avro then maps the stored data into the target schema by using the stored schema. Most avro libraries expect you to get the target schema from a separate source before reading data.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#44

Post by wyc »

kyllo wrote: post #25
leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

Real decimal type - invaluable when working with currency What does JavaScript do with this though, just cast it to a float?

I find that many financial technology companies opt to store currency as strings. The small overhead is typically well worth freedom from floating-point errors.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#45

Post by brandonbloom »

jcrites wrote: post #15
escherize wrote: post #3

Is there a source for benchmarks/reviews for the various ways to represent data? As far as I see it, there are a lot of them that I'd like to hear pros/cons for: json, edn + transit (my fave), yaml, google protobufs, thrift (?), as well as Ion. And where does Ion fit here?

Ion's advantage is that it's both strongly-typed with a rich type system, as well as self-describing. Data formats like JSON and XML can be somewhat self-describing, but they aren't always completely. Both tend to need to embed more complex data types as either strings with implied formats, or nested structures. (Consider: How would you represent a timestamp in JSON such that an application could unambiguously read i…

EDN supports dates, etc, too.

However, it doesn't support blobs. I'm conflicted about this point. On one hand, small blobs can occasionally be useful to send within a larger payload. On the other hand, small blobs almost always become large blobs, and so I'd rather plan for out-of-band (preferably even content addressable) representations of blobs.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#46

Post by voltagex_ »

tantalor wrote: post #38
voltagex_ wrote: post #28

Open question to anyone reading this: Would you use Ion if you were designing a new house-wide message queue? (e.g. broadcast messages to /Home/Lounge/Lights/ to turn on/off)

No that's overkill, just use JSON.

This is offtopic, but I'm looking into having JSON schemas on another Mosquitto topic so that clients can request it, kinda like SOAP's WSDL (recovering C# programmer here).

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#48

Post by ddlatham »

wyc wrote: post #23
jcrites wrote: post #18

Earlier quoted context omitted.

What do you mean by they both have self-describing schemas? In order to read or write Avro data, an application needs to possess a schema for that data -- the specific schema that the data was written with, and (when writing) the same schema that a later reader expects to find. This means the data is not self-describing. Ion is designed to be self-describing, meaning that no schema is necessary to deserialize and int…

I think it depends on what level you're referring to. If you mean record-level, then I concede that it's not self-describing. However, looking at the suggested use cases, it seems that it's "self-describing" in that you'll always be able to decode data stored according to what the documentation recommends: "Avro data is always serialized with its schema. Files that store Avro data should always also include the schem…

There are some use cases where record-level self description is very useful. For example when dealing with small records in a database or NoSQL store or message queue that could be written by multiple versions of applications. To cover that case well with Avro where records are not self describing really requires something like a schema registry and embedding a schema id with each record (e.g. http://www.confluent.io/blog/schema-registry-kafka-stream-pr... ).

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#49

Post by QuercusMax »

saosebastiao wrote: post #32

Do any of the popular message serialization formats have first class support for algebraic data types? It seems like every one I've researched has to be hacked in some way to provide for sum types.

Protocol buffers support oneof, which is a union type. https://developers.google.com/protocol-buffers/docs/proto#on...

(Insert joke here about Google engineers just copying around protobufs.)

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#50

Post by nikolay »

leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

Okay, but they did a really poor job marketing it in this release. Plus, if it's used within Amazon, why it's Java-only so far?
Post reply on HN