Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What even is a JSON number?

blog.trl.sn

Post reply on HN
Print view View original

71–80 of 151 posts

Re: What even is a JSON number?

#71

Post by greggyb »

golergka wrote: post #66
greggyb wrote: post #65

Earlier quoted context omitted.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

We're talking about deserialising JSONs in the application server here, nobody stops you from treating ids as numbers on the database side of things. But also, this sounds like a premature optimisation. Most applications will never reach a level where their performance is actually impacted by string comparison, and when you reach that stage, you're likely have already thrown out a lot of other common sense stuff like…

I work primarily in data analytics. It tends to become noticeable in my experience as soon as you're at a few million records[0] on at least one side of a relationship. Especially as we see more columnar databases in analytics, the row count accounts for more than total data size for this sort of thing.

Due to the type of aggregate queries that typify analytics workloads, almost everything turns into a scan, whether it be of the a table, field, or index. Strings occupy more space on disk, or in RAM, so scanning a whole column or table simply takes longer, because you have to shovel more bytes through the CPU. This doesn't even take into account the relative CPU time to actually do the comparisons.

I've never personally worked with a system that has string keys shorter than 10 [1][2] characters. At that point, regardless of how you pack characters into a register, you're occupying more bits with two strings of character data than you would with two 64-bit integers[3]. This shows through in join time.

[0]: Even modestly sized companies tend to have at least a few tables that get into the millions of records.

[1]: I've heard of systems with shorter string keys

[2]: Most systems with string keys I've encountered have more than 10 characters.

[3]: The vast majority of systems I've seen since the mid-2010s use 64-bit integers for keys for analytics. 32-bit integers seemed to phase out for new systems I've seen since ~2015, but were more common prior to that.

Re: What even is a JSON number?

#72

Post by Recursing »

hinkley wrote: post #42

One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…

> You have about a .1% chance of generating an ID that gets truncated in JavaScript

I don't follow. 1-(Number.MAX_SAFE_INTEGER / 2*63) ~ 99.9%, so don't you have a >99% chance of generating an ID that gets truncated in js?

Re: What even is a JSON number?

#73

Post by hugh-avherald »

paulddraper wrote: post #2

> I-JSON messages SHOULD NOT include numbers that express greater magnitude or precision than an IEEE 754 double precision number provides I'm confused by this. What is the precision of 0.1, relative to IEEE 754? If I read it correctly, that statement is saying: json_number_precision(json_number) ^ How do I calculate these values?

I think the spec is saying that it is the message that should not express greater magnitude or precision, not 'the number'.

So including the string "0.1" in a message is fine because v = 0.1 implies 0.05 < v < 0.15, but including 0.100000000000000000000000000000000000 would not be.

Re: What even is a JSON number?

#74

Post by js2 »

Recursing wrote: post #72
hinkley wrote: post #42

One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…

> You have about a .1% chance of generating an ID that gets truncated in JavaScript I don't follow. 1-(Number.MAX_SAFE_INTEGER / 2*63) ~ 99.9%, so don't you have a >99% chance of generating an ID that gets truncated in js?

IEEE 754 can represent integers larger than MAX_SAFE_INTEGER, just not all of them:

https://en.wikipedia.org/wiki/Double-precision_floating-poin...

That's still going to be a greater than 0.1% chance of hitting a non-representable value though.

Re: What even is a JSON number?

#75

Post by yawaramin »

cerved wrote: post #24
Izkata wrote: post #20

Earlier quoted context omitted.

This is why Decimal exists: Python 3.8.10 (default, Nov 22 2023, 10:22:35) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from decimal import Decimal >>> Decimal('100000.000000000017') Decimal('100000.000000000017') For example: >>> import json >>> json.loads('{"a": 100000.000000000017}') {'a': 100000.00000000001} >>> json.loads('{"a": 100000.000000000017}', parse_floa…

but serializing/deserializing decimal using the json module is futile

Why is it futile? It can be serialized/deserialized perfectly through its string representation.

Re: What even is a JSON number?

#77

Post by hinkley »

js2 wrote: post #74
Recursing wrote: post #72

Earlier quoted context omitted.

> You have about a .1% chance of generating an ID that gets truncated in JavaScript I don't follow. 1-(Number.MAX_SAFE_INTEGER / 2*63) ~ 99.9%, so don't you have a >99% chance of generating an ID that gets truncated in js?

IEEE 754 can represent integers larger than MAX_SAFE_INTEGER, just not all of them: https://en.wikipedia.org/wiki/Double-precision_floating-poin... That's still going to be a greater than 0.1% chance of hitting a non-representable value though.

It’s been a long long time. I may be remembering the ratio wrong, or we might have been clipping the range a bit.

Re: What even is a JSON number?

#78

Post by hinkley »

golergka wrote: post #66
greggyb wrote: post #65

Earlier quoted context omitted.

Until you want faster joins, in which case, comparisons of integers tend to be much faster on hardware I am aware of than string comparisons.

We're talking about deserialising JSONs in the application server here, nobody stops you from treating ids as numbers on the database side of things. But also, this sounds like a premature optimisation. Most applications will never reach a level where their performance is actually impacted by string comparison, and when you reach that stage, you're likely have already thrown out a lot of other common sense stuff like…

Mostly a deal of defaults on our stack. Tweaked a couple of things in a few places to stop the bleeding. Then had to fix all of the tests.

Re: What even is a JSON number?

#79

Post by int_19h »

olejorgenb wrote: post #45
egwor wrote: post #30

I think the thing folk miss is when there’s an error like divide by zero, or the calculation would return NaN. I feel like this is the main gap/concern with using JSON and it seems to be rarely discussed.

Agreed, this can be a pain. Python by default serialize and de-serialize the `NaN` literal, making you pay some cleanup cost once you need to interopt with other systems. (same for `Inf`) Say what you want about NaN, but IEEE 754 is the facto way of dealing with floating points in computers and even if NaNs and Infs are a bit "fringe" it's unfortunate that the most popular serialization format can not represent these…

There are so many things that are poorly thought out or underspecified in JSON, it's amazing that it got so widely adopted for interop. No wonder that it became a perpetual source of serialization bugs.

Re: What even is a JSON number?

#80

Post by int_19h »

lofenfew wrote: post #63
nigeltao wrote: post #34

When I wrote my jsonptr tool a few years ago, I noticed that some JSON libraries (in both C++ and Rust) don't even do "parse a string of decimal digits as a float64" properly. I don't mean that in the "0.3 isn't exactly representable; have 0.30000000000000004 instead" sense. I mean that rapidjson (C++) parsed the string "0.99999999999999999" as the number 1.0000000000000003. Apart from just looking weird, it's a diff…

this requires multiple precision to do properly and isn't useful most of the time. its odd to describe this as "not properly". you might say "with exact rounding", but that makes it clearer that this isn't that useful a feature, especially since we usually expect floats to be inexact in the first place.

With JSON, there's essentially no such thing as "properly" when it comes to parsing numbers, since the spec doesn't limit the ability of the implementation to constrain width and precision. It only says that float64 is common and therefore "good interoperability can be achieved by implementations that expect no more precision or range than these provide", but note the complete absence of any guarantees in that wording.

The only sane thing with JSON is to avoid numbers altogether and just use decimal-encoded strings. This forces the person parsing it on the other end to at least look up the actual limits defined by your schema.

Post reply on HN