Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

151–160 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#151
post #15

Is there a source for benchmarks/reviews for the various ways to represent data? As far as I see it, there are a lot of them that I'd like to hear pros/cons for: json, edn + transit (my fave), yaml, google protobufs, thrift (?), as well as Ion. And where does Ion fit here?

Ion's advantage is that it's both strongly-typed with a rich type system, as well as self-describing. Data formats like JSON and XML can be somewhat self-describing, but they aren't always completely. Both tend to need to embed more complex data types as either strings with implied formats, or nested structures. (Consider: How would you represent a timestamp in JSON such that an application could unambiguously read i…

> Another downside is that it's difficult to isolate Ion-aware code from the more general purpose "business logic" in an application, due to the absence of a serialization layer producing/consuming POJOs; instead it's common to read an Ion structure from the wire and access it directly from application logic.

This is indeed a common pitfall, especially since traversing Ion is slow and expensive. I've squeezed up to 30% performance gain by converting Ion data to POJOs up front and just using those.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#153

Are there any object marshalling/serialization solution for Ion? (Like GSON, Jackson)

It _is_ possible to adapt Jackson (with minimal effort) to use Ion, since it's very similar to Jackson's native JSON format.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#154
Things I dislike about Ion, having used it while at Amazon:

- IonValues are mutable by default. I saw bugs where cached IonValues were accidentally changed, which is easy to do: IonSequence.extract clears the sequence [1], adding an IonValue to a container mutates the value (!) [2], etc.

- IonValues are not thread-safe [3]. You can call makeReadOnly() to make them immutable, but then you'll be calling clone since doing anything useful (like adding it to a list) will need to mutate the value. While it says IonValues are not even thread-safe for reading, I believe this is not strictly true. There was an internal implementation that would lazily materialize values on read, but it doesn't look like it's included in the open source version.

- IonStruct can have multiple fields with the same name, which means it can't implement Map. I've never seen anyone use this (mis)feature in practice, and I don't know where it would be useful.

- Since IonStruct can't implement Map, you don't get the Java 8 default methods like forEach, getOrDefault, etc.

- IonStruct doesn't implement keySet, values, spliterator, or stream, and thus doesn't play well with the Java 8 Stream API.

- Calling get(fieldName) on an IonStruct returns null if the field isn't present. But the value might also be there and be null, so you end up having to do a null check AND call isNullValue(). I'm not convinced it's a worthwhile distinction, and would have preferred a single way of doing it. You can already call containsKey to check for the presence of a field.

- In practice most code that dealt with Ion was nearly as tedious and verbose as pulling values out of an old-school JSONObject. Every project seemed to have a slightly different IonUtils class for doing mundane things like pulling values out of structs, doing all the null checks, casting, etc. There was some kind of adapter for Jackson that would allow you to deserialize to a POJO, but it didn't seem like it was widely used.

[1] https://github.com/amznlabs/ion-java/blob/master/src/softwar...

[2] https://github.com/amznlabs/ion-java/blob/master/src/softwar...

[3] https://github.com/amznlabs/ion-java/blob/master/src/softwar...

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#155

Earlier quoted context omitted.

You are choosing a definition of strong typing that supports your argument, but the argument is over the meaning of strong typing to begin with. It's not as if there's some universally accepted definition of strong typing. Like functional programming, functional purity, object oriented, etc.—none of these terms are universally defined.

The fact that "strong typing" has no universal definition is exactly why I think it's not useful.

I hate feeling like I'm nitpicking, but I don't think that's true. I think they do have a well-accepted definition, which appears in Wikipedia, in assorted articles online, and in computer science publications. Here are some examples of CS publications that describe a research contribution in terms of strong typing:

> Strong typing of object-oriented languages revisited. This paper is concerned with the relation between subtyping and subclassing and their influence on programming language design. [...] The type system of a language can be characterized as strong or weak and the type checking mechanism as static or dynamic. http://dl.acm.org/citation.cfm?id=97964

> GALILEO: a strongly-typed, interactive conceptual language. Galileo, a programming language for database applications, is presented. Galileo is a strongly-typed, interactive programming language designed specifically to support semantic data model features (classification, aggregation, and specialization), as well as the abstraction mechanisms of modern programming languages (types, abstract types, and modularization). http://dl.acm.org/citation.cfm?id=3859

> Design and implementation of an object-oriented strongly typed language for distributed applications. http://dl.acm.org/citation.cfm?id=99813

> Strongly typed heterogeneous collections. (Oleg Kiselyov et al.) http://dl.acm.org/citation.cfm?id=1017488

> Strongly typed genetic programming. Genetic programming is a powerful method for automatically generating computer programs via the process of natural selection [but] there is no way to restrict the programs it generates to those where the functions operate on appropriate data types. [When] programs manipulate multiple data types and contain functions designed to operate on particular data types, this can lead to unnecessarily large search times and/or unnecessarily poor generalization performance. Strongly typed genetic programming (STGP) is an enhanced version of genetic programming that enforces data-type constraints and whose use of generic functions and generic data types makes it more powerful than other approaches to type-constraint enforcement http://dl.acm.org/citation.cfm?id=1326695

The argument that the terms have no universal definition cannot be sound in light of their widespread use in computer science publications, even in the title and abstract. Perhaps what you mean to say is that the terms don't have a completely unambiguous or formal definition. That's probably true, but not all CS terms do. The words are contextual and exist on a spectrum, in the sense that a strongly-typed thing is typically in comparison to a more-weakly-typed thing [1]. However, the fact that they're widely used by CS researchers is why I think we should reject the argument that they don't have a universal definition or are not useful. CS researchers like Oleg Kiselyov use the term when describing their papers and characterizing their contributions.

[1] This is true for static and dynamic typing as well: they exist in degrees. Rust can verify type proofs that other languages can't regarding memory safety. Some languages can verify that integer indexes into an array won't go out of bounds. Thus it's not the case that a given language is either statically typed or dynamically typed; rather, each aspect of how it works can be characterized on a spectrum from statically verified to dynamically verified.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#156

Earlier quoted context omitted.

The fact that "strong typing" has no universal definition is exactly why I think it's not useful.

I hate feeling like I'm nitpicking, but I don't think that's true. I think they do have a well-accepted definition, which appears in Wikipedia, in assorted articles online, and in computer science publications. Here are some examples of CS publications that describe a research contribution in terms of strong typing: > Strong typing of object-oriented languages revisited. This paper is concerned with the relation betw…

> I think they do have a well-accepted definition [...] [You] shouldn't confuse your dislike for them for the absence of a well-accepted definition that's widely used in computer science literature.

Just upthread, you said:

> The notions of "strong" and "weak" typing have never been particularly well-defined

And the Wikipedia article you cited (https://en.wikipedia.org/wiki/Strong_and_weak_typing) says:

> These terms do not have a precise definition

The Wikipedia article also says:

> A number of different language design decisions have been referred to as evidence of "strong" or "weak" typing. In fact, many of these are more accurately understood as the presence or absence of type safety, memory safety, static type-checking, or dynamic type-checking.

Also on Wikipedia (https://en.wikipedia.org/wiki/Type_system):

> Languages are often colloquially referred to as "strongly typed" or "weakly typed". In fact, there is no universally accepted definition of what these terms mean. In general, there are more precise terms to represent the differences between type systems that lead people to call them "strong" or "weak".

...which is exactly what I'm saying in this entire thread.

It's very strange to me how you seem really seem to want other people to be on board with your particular interpretation of what everybody (even you, 13 hours ago) agrees is not a very well-defined concept.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#157

Earlier quoted context omitted.

The fact that "strong typing" has no universal definition is exactly why I think it's not useful.

I hate feeling like I'm nitpicking, but I don't think that's true. I think they do have a well-accepted definition, which appears in Wikipedia, in assorted articles online, and in computer science publications. Here are some examples of CS publications that describe a research contribution in terms of strong typing: > Strong typing of object-oriented languages revisited. This paper is concerned with the relation betw…

> This is true for static and dynamic typing as well: they exist in degrees. Rust can verify type proofs that other languages can't regarding memory safety. Some languages can verify that integer indexes into an array won't go out of bounds. Thus it's not the case that a given language is either statically typed or dynamically typed

Memory safety and static/dynamic typing are orthogonal. C is statically typed but memory unsafe. Rust is statically typed but memory safe (except in unsafe blocks). Lua is dynamically-typed but memory safe.

I agree that it's possible to mix elements of static and dynamic typing in a single language. C++ is generally statically typed, but also supports dynamic_cast.

But generally speaking, static and dynamic typing have a very precise definition. Something that carries around type information at runtime is dynamically typed. Something that does type analysis at compile time so that the runtime doesn't need to carry type information is statically typed.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#158
post #2

This reminds me a lot of Avro: https://avro.apache.org/docs/current/ They both have self-describing schemas, support for binary values, JSON-interoperability, basic type systems (Ion seems to support a few more field types), field annotations, support for schema evolution, code generation not necessary, etc. I think Avro has the additional advantages of being production-tested in many different companies, a fully-JSO…

Amazon invented Ion because yaml, Avro, etc. didn't exist at the time. Ion is actually pretty old. The timing of open-sourcing it mystifies me a bit. Maybe Amazon is trying to become more open-source friendly, like Microsoft did? Perhaps more likely: they're planning on making some internal APIs that use ION heavily public?

[deleted]

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#159
post #65

Earlier quoted context omitted.

BSON is awful. * It doesn't have "true" types in the sense that Ion does. It's basically just a binary serialization of JSON, with extra stuff. * Despite being a binary format, it's actually bulkier than JSON in most situations. * It removes any semblance of canonicity from many representations. A number, for instance, can potentially be represented by any of at least 3 types (double, int32, and int64). * It has sign…

Most of this comes from BSON also being the internal storage format for a database server. For example, at least the redundant string NULs make it possible to use C library functions without copying, the unpacked ints allow direct dereferencing, etc. I've no clue about the trailing NUL on the record itself, perhaps a safety feature?

> I've no clue about the trailing NUL on the record itself, perhaps a safety feature?

Could be. Or perhaps there's enough code paths in common between string parsing and document parsing that they decided to put a trailing null byte on both.

Stepping back a bit, though, the fact that BSON is optimized for "direct" use in C code is really scary. That suggests that any failure to completely validate BSON data could open up vulnerabilities in C code manipulating it.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#160

Earlier quoted context omitted.

I'm sure there are ion bindings for every language in common use at Amazon. But a huge percentage of Amazon code is Java, so presumably this one was the best maintained and documented.

I doubt it, when I was there Ion was only used by only a handful of Java teams doing backend work. It was also horribly documented and supported at the time (3.5 years ago).

I left < a year ago and never heard of ion.
Post reply on HN