Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What even is a JSON number?

blog.trl.sn

Post reply on HN
Print view View original

51–60 of 151 posts

Re: What even is a JSON number?

#51

Post by js2 »

bterlson wrote: post #4
EdSchouten wrote: post #3

I think the description for Go is inaccurate/incomplete. You can call this function to instruct the decoder to leave numbers in unparsed string form: https://pkg.go.dev/encoding/json#Decoder.UseNumber That allows you to capture/forward numbers without any loss of precision.

I have added this note, thanks! In the blog I am mostly trying to show the behavior you get using the (maybe defacto) stdlib with its default configuration, but this is useful data to call out.

If you're going to extend Go the courtesy of customizing the parser, oughtn't you do the same for Python (and all the languages)?

To wit, Python's json module has `parse_float` and `parse_int` hooks:

https://docs.python.org/3/library/json.html#encoders-and-dec...

Example:

  >>> json.loads('{"int":12345,"float":123.45}', parse_int=str, parse_float=str)
  {'int': '12345', 'float': '123.45'}
FWIW, when I've cared about interop and controlled the schema, I've specified JSON strings for numbers, along with the range, precision, and representation. This is no worse (nor better) than using RFC 3339 for dates.

Re: What even is a JSON number?

#52

Post by vbezhenar »

My opinion is that a safe approach is to use either 52-bit integer number or 64-bit floating number to keep JavaScript compatibility. JavaScript is too important and at the same time, the errors are too terrific (JS will silently round to the nearest 52-bit integer number which could lead to various exploits) to skip on that. If you need anything else, just use strings.

Re: What even is a JSON number?

#53

Post by Someone »

Other values one could test for:

- “+1” (not a valid number, according to ECMA-404 and RFC-8259)

- “+0” (also not a valid number, but trickier than “+1” because IEEE floats have “+0” and “-0”)

- “070” (not a valid number, but may get parsed as octal 56)

- “1.” (not a valid number in json)

- “.1” (not a valid number in json)

- “0E-0” (a valid number in json)

There probably are others.

Re: What even is a JSON number?

#54

Post by akubera »

kccqzy wrote: post #28

I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…

I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the complexity is shifted into checking for NaNs, which I've been trying to avoid.

Sounds like Haskell made the right call: put warnings in the docs and steer the user in the right direction. Keeps implementation simple and users in control.

To the point of the article, serde_json support is improving in the next version of BigDecimal, so you'll be able to decorate your BigDecimal fields and it'll parse numeric fields from the JSON source, rather than json -> f64 -> BigDecimal.

    #[derive(Serialize, Deserialize)]
    pub struct MyStruct {
      #[serde(with = "bigdecimal::serde::json_num")]
      value: BigDecimal,
    }
Whether or not this is a good idea is debatable[^], but it's certainly something people have been asking for.

[^] Is every part of your system, or your users' systems, going to parse with full precision?

Re: What even is a JSON number?

#55

Post by your_fin »

apantel wrote: post #14
ape4 wrote: post #6

Since JSON is so widely used it should be modified to support more types - Mongo DB's Extended JSON supports all the BSON (Binary) types: Array Binary Date Decimal128 Document Double Int32 Int64 MaxKey MinKey ObjectId Regular Expression Timestamp https://www.mongodb.com/docs/manual/reference/mongodb-extend...

Much more valuable than any such extension would be a way to annotate types and byte lengths of keys and values so that parsers could work more efficiently. I’ve spent a lot of time making a fast JSON parser in Java and the thing that makes it so hard is you don’t know how many bytes anything is, or what type. It’s hard to do better than naive byte-by-byte parsing.

If you control the underlying data, I must reccomend Amazon Ion! Its text format is a strict superset of JSON, but they also maintain binary format that will round-trip data and is designed for efficient scanning. There's even prefixed annotations if you want them :)

It also specs proper decimal values, mitigating the issues presented in the OP.

https://amazon-ion.github.io/ion-docs/

Re: What even is a JSON number?

#56

Post by NegativeLatency »

secondcoming wrote: post #50
hinkley wrote: post #42

One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…

Why would the value get truncated?

not all numbers are representable as the particular type of floating point number that js uses

nice pics here: https://en.wikipedia.org/wiki/Floating-point_arithmetic

Re: What even is a JSON number?

#57

Post by bterlson »

js2 wrote: post #51
bterlson wrote: post #4

Earlier quoted context omitted.

I have added this note, thanks! In the blog I am mostly trying to show the behavior you get using the (maybe defacto) stdlib with its default configuration, but this is useful data to call out.

If you're going to extend Go the courtesy of customizing the parser, oughtn't you do the same for Python (and all the languages)? To wit, Python's json module has `parse_float` and `parse_int` hooks: https://docs.python.org/3/library/json.html#encoders-and-dec... Example: >>> json.loads('{"int":12345,"float":123.45}', parse_int=str, parse_float=str) {'int': '12345', 'float': '123.45'} FWIW, when I've cared about inte…

I'm just a JS guy trying to understand the world around me and documenting what I find, not trying to be discourteous (or even courteous). I'll add the note about Python, thanks for calling it out. FWIW JS does not have a similar capability so I can't add a note there.

Re: What even is a JSON number?

#58

Post by Dylan16807 »

ape4 wrote: post #6

Since JSON is so widely used it should be modified to support more types - Mongo DB's Extended JSON supports all the BSON (Binary) types: Array Binary Date Decimal128 Document Double Int32 Int64 MaxKey MinKey ObjectId Regular Expression Timestamp https://www.mongodb.com/docs/manual/reference/mongodb-extend...

JSON is not the place to be so fussy about number widths, and things like MaxKey and 24-hex-value ObjectId would be ridiculous.

Re: What even is a JSON number?

#59

Post by AaronFriel »

akubera wrote: post #54
kccqzy wrote: post #28

I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…

I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the com…

I'd strongly recommend against this default - it's a major blocker for using the Haskell library with web APIs as it transforms JSON RPC into into readily available denial of service attacks.

8 billion digits (~100 bits?) is far more than should be used.

Would it possible to use const generics to expose a `BigDecimal` or `BigDecimal` type with bounded precision for serde, and disallow this unsafe `BigDecimal` entirely?

If not, I expect BigDecimal will be flagged in a CVE in the near future for causing a denial of service.

Re: What even is a JSON number?

#60

Post by js2 »

bterlson wrote: post #57
js2 wrote: post #51

Earlier quoted context omitted.

If you're going to extend Go the courtesy of customizing the parser, oughtn't you do the same for Python (and all the languages)? To wit, Python's json module has `parse_float` and `parse_int` hooks: https://docs.python.org/3/library/json.html#encoders-and-dec... Example: >>> json.loads('{"int":12345,"float":123.45}', parse_int=str, parse_float=str) {'int': '12345', 'float': '123.45'} FWIW, when I've cared about inte…

I'm just a JS guy trying to understand the world around me and documenting what I find, not trying to be discourteous (or even courteous). I'll add the note about Python, thanks for calling it out. FWIW JS does not have a similar capability so I can't add a note there.

Fair enough! Thank you for the writeup.
Post reply on HN