Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

github.com

Post reply on HN
Print view View original

101–110 of 174 posts

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#101

Post by buro9 »

kyllo wrote: post #25
leef wrote: post #11

Finally! I've had to live the JSON nightmare since I left Amazon. Some of the benefits over JSON: * Real date type * Real binary type - no need to base64 encode * Real decimal type - invaluable when working with currency * Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo'). * Text and binary format * Symbol ta…

Real decimal type - invaluable when working with currency What does JavaScript do with this though, just cast it to a float?

The real way is:

  "price": {
    "amount": "1500",
    "scale": 2,
    "symbol": "GBP",
  }
Currency has 3 properties, the amount, scale, and symbol.

Amount is a string, it holds a bigint. Yes, it's a string.

The value of Scale can be up to 5 but is usually 2 or 3.

Symbol is the ISO code.

Whenever I see a financial system that uses "amount": 15.00 I know that the system is ill-conceived.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#102

Post by tlarkworthy »

Cshelton wrote: post #92
justin_oaks wrote: post #86

Earlier quoted context omitted.

Do you really need those extra 11 bits? Javascript numbers accurately represent integers up to 2^53 - 1. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

I just did some ownership percentage stuff where it's not uncommon to go 16 decimal places out...working with JavaScript on this was a pain. Never thought I'd care about that .00000000000001 difference hah...

floating point works best near 0, most of the numbers it can represent lie near 0 (negative exponent is a number less than 1).

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#103

Post by kyllo »

troyk wrote: post #34
kyllo wrote: post #25

Earlier quoted context omitted.

Real decimal type - invaluable when working with currency What does JavaScript do with this though, just cast it to a float?

move the decimal and use an int (cents in US). It still blows me away that javascript has become so popular on the server without 64bit int types.

JavaScript doesn't have any int types at all. Your solution works until you need to do division...

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#104

Post by kyllo »

wyc wrote: post #44
kyllo wrote: post #25

Earlier quoted context omitted.

Real decimal type - invaluable when working with currency What does JavaScript do with this though, just cast it to a float?

I find that many financial technology companies opt to store currency as strings. The small overhead is typically well worth freedom from floating-point errors.

Don't they have to convert it back to a float to do math operations on it (if they're using JavaScript)?

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#105

Post by Dylan16807 »

snuxoll wrote: post #93
justin_oaks wrote: post #86

Earlier quoted context omitted.

Do you really need those extra 11 bits? Javascript numbers accurately represent integers up to 2^53 - 1. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Until you need to deal with decimals instead of floats, then you are going to hate yourself because you have to pull in some third party library because the language treats every single number as a float (and floating point errors are a lot more common than most people think even when they are adding together simple numbers).

Integers will do no better at pretending to be a decimal type without a library.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#106

Post by pjmlp »

JonathonW wrote: post #89
conradev wrote: post #22

Earlier quoted context omitted.

Sounds a lot like Apple's property list format, which shares almost everything you listed in common, except for annotations and symbol tables. Its binary format was introduced in 2002! Edit: Property lists only support integers up to 128 bits in size and double-precision floating point numbers. On top of those, Ion also supports infinite precision decimals.

Plists are nifty, but the text format's XML-based, which makes it too complex and too verbose to be a general-purpose alternative to something like JSON. (plutil "supports" a json format, but it's not capable of expressing the complete feature set of the XML or binary formats.)

I don't get this gripe with XML, it is meant to be used by tools not to be written by hand.

Where is the XPath and XQuery for JSON?

Do people really think that manually iterating over the whole JSON document to find the data or writing yet another parser, is better?

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#107

Post by [deleted] »

duskwuff wrote: post #51
vparikh wrote: post #30

Wasn't this solved already by the BSON specification - http://bsonspec.org ? Sure this allows you a definition of types, but this could easily be done using standard JSON meta data for each field. I find BSON simpler and more elegant.

BSON is awful. * It doesn't have "true" types in the sense that Ion does. It's basically just a binary serialization of JSON, with extra stuff. * Despite being a binary format, it's actually bulkier than JSON in most situations. * It removes any semblance of canonicity from many representations. A number, for instance, can potentially be represented by any of at least 3 types (double, int32, and int64). * It has sign…

[deleted]

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#108

Post by deathanatos »

I can't decide if "JSON-superset" is technically accurate or not.

JSON's string literals come from JavaScript, and JavaScript only sortof has a Unicode string type. So the \u escape in both languages encodes a UTF-16 code unit, not a code point. That means in JSON, the single code point U+1f4a9 "Pile of Poo" is encoded thusly:

    "\ud83d\udca9"
JSON specifically says this, too,

   Any character may be escaped.  If the character is in the Basic
   Multilingual Plane (U+0000 through U+FFFF), then it may be
   represented as a six-character sequence: a reverse solidus, followed
   by the lowercase letter u, followed by four hexadecimal digits that
   encode the character's code point.  The hexadecimal letters A though
   F can be upper or lowercase.  So, for example, a string containing
   only a single reverse solidus character may be represented as
   "\u005C".

   [… snip …]

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".
Now, Ion's spec says only:

   U+HHHH	\uHHHH	4-digit hexadecimal Unicode code point
But if we take it to mean code point, then if the value is a surrogate… what should happen?

Looking at the code, it looks like the above JSON will parse:

  1. Main parsing of \u here:
     https://github.com/amznlabs/ion-java/blob/1ca3cbe249848517fc6d91394bb493383d69eb61/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java#L2429-L2434

  2. which is called from here, and just appended to a StringBuilder:
     https://github.com/amznlabs/ion-java/blob/1ca3cbe249848517fc6d91394bb493383d69eb61/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java#L1975
My Java isn't that great though, so I'm speculating. But I'm not sure what should happen.

This is just one of those things that the first time I saw it in JSON/JS… a part of my brain melted. This is all a technicality, of course, and most JSON values should work just fine.

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#109

Post by jon-wood »

pjmlp wrote: post #106
JonathonW wrote: post #89

Earlier quoted context omitted.

Plists are nifty, but the text format's XML-based, which makes it too complex and too verbose to be a general-purpose alternative to something like JSON. (plutil "supports" a json format, but it's not capable of expressing the complete feature set of the XML or binary formats.)

I don't get this gripe with XML, it is meant to be used by tools not to be written by hand. Where is the XPath and XQuery for JSON? Do people really think that manually iterating over the whole JSON document to find the data or writing yet another parser, is better?

http://jmespath.org/

Re: Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset

#110

Post by jrgv »

deathanatos wrote: post #108

I can't decide if "JSON-superset" is technically accurate or not. JSON's string literals come from JavaScript, and JavaScript only sortof has a Unicode string type. So the \u escape in both languages encodes a UTF-16 code unit , not a code point . That means in JSON, the single code point U+1f4a9 "Pile of Poo" is encoded thusly: "\ud83d\udca9" JSON specifically says this, too, Any character may be escaped. If the cha…

> But if we take it to mean code point, then if the value is a surrogate… what should happen?

Surrogates are code points. The spec does not say what should happen if the surrogate is invalid (for example, if only the first surrogate of a surrogate pair is present), but neither does the JSON spec.

Java internally also represents non-BMP code points using surrogates. So, simply appending the surrogates to the string should yield a valid Java string if the surrogates in the input are valid.

Post reply on HN