Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What even is a JSON number?

blog.trl.sn

Post reply on HN
Print view View original

61–70 of 151 posts

Re: What even is a JSON number?

#61

Post by NateEag »

hn_throwaway_99 wrote: post #43
whalesalad wrote: post #12

I tend to end up encoding everything as an integer (multiply by 1000, 10000 etc) and then turn it back into a float/decimal on decode. For instance if I am building a system dealing with dollar amounts I will store cent amounts everywhere, communicate cent amounts over the wire, etc. then treat it as a presentation concern to render it as a dollar amount.

The problem with that (which I have seen in practice) is that you are essentially hard coding the maximum precision you will accept for every client that needs to interpret your JSON. For example, you say you store monetary amounts as cents. What if you needed to store US gas prices, which are normally priced in amounts ending in 9/10ths of a cent? If you want to keep your values as integers you need to change your p…

This is a good point.

It's not widely known, but US gasoline prices are actually in a defined currency unit, the mill (https://en.m.wikipedia.org/wiki/Mill_(currency)).

For most purposes, using mills as the base unit would be sufficient resolution.

Re: What even is a JSON number?

#62

Post by hatthew »

secondcoming wrote: post #50
hinkley wrote: post #42

One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…

Why would the value get truncated?

FTA: JavaScript's built-in JSON implementation is limited to the range and precision of a double.

Obviously, not all int64 values are representable in float64 (double).

Re: What even is a JSON number?

#63

Post by lofenfew »

nigeltao wrote: post #34

When I wrote my jsonptr tool a few years ago, I noticed that some JSON libraries (in both C++ and Rust) don't even do "parse a string of decimal digits as a float64" properly. I don't mean that in the "0.3 isn't exactly representable; have 0.30000000000000004 instead" sense. I mean that rapidjson (C++) parsed the string "0.99999999999999999" as the number 1.0000000000000003. Apart from just looking weird, it's a diff…

this requires multiple precision to do properly and isn't useful most of the time. its odd to describe this as "not properly". you might say "with exact rounding", but that makes it clearer that this isn't that useful a feature, especially since we usually expect floats to be inexact in the first place.

Re: What even is a JSON number?

#64

Post by marcosdumay »

akubera wrote: post #54
kccqzy wrote: post #28

I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…

I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the com…

It's best if your parser fails.

Serde has an interface that allows failing. That one should fail. There is also another that panics, and AFAIK it will automatically panic on any parser that fails.

Do not try to handle huge values, do not pretend your parser is total, and do not pretend it's a correct value.

If you want to create an specialized parser that handles huge numbers, that's great. But any general one must fail on them.

Re: What even is a JSON number?

#65

Post by greggyb »

golergka wrote: post #47
hinkley wrote: post #42

One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…

I've been burned by a similar issue too. Lesson here is never to use numbers for things you are not planning to do math on. Ids should always be strings.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

Re: What even is a JSON number?

#66

Post by golergka »

greggyb wrote: post #65
golergka wrote: post #47

Earlier quoted context omitted.

I've been burned by a similar issue too. Lesson here is never to use numbers for things you are not planning to do math on. Ids should always be strings.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

We're talking about deserialising JSONs in the application server here, nobody stops you from treating ids as numbers on the database side of things.

But also, this sounds like a premature optimisation. Most applications will never reach a level where their performance is actually impacted by string comparison, and when you reach that stage, you're likely have already thrown out a lot of other common sense stuff like db normalisation to get there, and we shouldn't judge "regular people" advice because it doesn't usually apply to you anyway.

Out of curiosity, have you ever seen an application that was meaningfully impacted by this? How gigantic was it?

----

Scratch that. I've actually thought about it some more, and now I'm not 100% sure it's premature, I have to investigate further to be sure. Question still stands though.

Re: What even is a JSON number?

#67

Post by cduzz »

greggyb wrote: post #65
golergka wrote: post #47

Earlier quoted context omitted.

I've been burned by a similar issue too. Lesson here is never to use numbers for things you are not planning to do math on. Ids should always be strings.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

Don't care.

A numeric identity is an identity and so is a string.

If you want to math it, it is a number, otherwise... string.

"Will you ever want the 95th percentile PID? Then it is not a number. Move on."

Re: What even is a JSON number?

#68

Post by kccqzy »

akubera wrote: post #54
kccqzy wrote: post #28

I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…

I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the com…

I think Haskell's warning-in-the-doc approach is not strong enough. I'd be in favor of distinguishing small and huge values using the type system. Have a Rust enum that contains either a small-ish number (the absolute value being 10^100 or less, but the threshold should be configurable preferably as a type parameter) or a huge number. Then the user will be required to handle it. Most of the time the user does not want huge numbers, so they will fail the parse explicitly when they do a match and find it.

Re: What even is a JSON number?

#69

Post by kevin_thibedeau »

cduzz wrote: post #67
greggyb wrote: post #65

Earlier quoted context omitted.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

Don't care. A numeric identity is an identity and so is a string. If you want to math it, it is a number, otherwise... string. "Will you ever want the 95th percentile PID? Then it is not a number. Move on."

Double precision floats can't represent every 64-bit integer. If you want to math it, what kind of number will you accept?

Re: What even is a JSON number?

#70

Post by felixfbecker »

hn_throwaway_99 wrote: post #43
whalesalad wrote: post #12

I tend to end up encoding everything as an integer (multiply by 1000, 10000 etc) and then turn it back into a float/decimal on decode. For instance if I am building a system dealing with dollar amounts I will store cent amounts everywhere, communicate cent amounts over the wire, etc. then treat it as a presentation concern to render it as a dollar amount.

The problem with that (which I have seen in practice) is that you are essentially hard coding the maximum precision you will accept for every client that needs to interpret your JSON. For example, you say you store monetary amounts as cents. What if you needed to store US gas prices, which are normally priced in amounts ending in 9/10ths of a cent? If you want to keep your values as integers you need to change your p…

and different currencies have different default precisions. So if you're dealing with multiple currencies, now you need both client and server to have a map of all currency precisions for formatting purposes that they agree on.

What's worse is that these things can also change over time and there is sometimes disagreement over what the canonical value is.

E.g. ISO 4217 (used by Safari, Firefox and NodeJS) will say that the Indonesian Rupiah (IDR) uses 2 decimal digits, while Unicode CLDR (used by Chrome) will say that they use 0 decimal digits. The former is the more "legalistic" definition, while the latter matches how people use the currency in reality.

This is not a real issue if you transfer amounts as decimal strings and then pass those to the Intl API for formatting (the formatting will just be different but still correct), but it's catastrophic if you use scaled-up integers (all amounts will be off by magnitudes).

For this reason I would always store currency amounts in an appropriate DECIMAL type in the DB and send currency amounts as strings over the wire.

Post reply on HN