81–90 of 151 posts
I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…
Earlier quoted context omitted.
Don't care. A numeric identity is an identity and so is a string. If you want to math it, it is a number, otherwise... string. "Will you ever want the 95th percentile PID? Then it is not a number. Move on."
Double precision floats can't represent every 64-bit integer. If you want to math it, what kind of number will you accept?
If you need to do math with the thing, use an appropriate type of number, of course.
It's weird that any parser that loses digits is tolerated. A parser that forces strings into uppercase US-ASCII never would be.
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
available and widely used, good interoperability can be achieved by
implementations that expect no more precision or range than these
provide, in the sense that implementations will approximate JSON
numbers within the expected precision. A JSON number such as 1E400
or 3.141592653589793238462643383279 may indicate potential
interoperability problems, since it suggests that the software that
created it expects receiving software to have greater capabilities
for numeric magnitude and precision than is widely available.
Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
I miss XML these days. Sure, it was verbose and had a bunch of different and probably excessive numeric types defined for XML Schema... but at least they were well-defined (https://www.w3.org/TR/xmlschema-2/#built-in-datatypes). And, on the other hand, without a schema, all you had were strings. Either way, no mismatched expectations.
Earlier quoted context omitted.
Makes sense for dollars, but for anything like graphics or physics I'd consider a power of two like 1,024 as the fixed-point factor instead. My intuition tells me that "x * 1000 / 1000 == x" might not be true for all numbers if you're using floats.
A sure sign of an inexperienced programmer in numerical computing is when they check for equality to zero of a floating-point number as if (x == 0) ... instead of something like if (abs(x) where eps is a suitably defined small number.
C The key here is that beta is likely to be an exact value that is entered as a constant, and detecting it allows for a worthwhile optimization.
One of the first Ajax projects I worked on was multi tenant, and someone decided to solve the industrial espionage problem by using random 64 bit identifiers for all records in the system. You have about a .1% chance of generating an ID that gets truncated in JavaScript, which is just enough that you might make it past MVP before anyone figures out it’s broken, and that’s exactly what happened to us. So we had to go…
I've been burned by a similar issue too. Lesson here is never to use numbers for things you are not planning to do math on. Ids should always be strings.
I'll add that for Haskell, the library everyone uses for JSON parses numbers into Scientific types with almost unlimited size and precision. I say almost unlimited because they use a decimal coefficient-and-exponent representation where the exponent is a 64-bit integer. The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine an…
How does it handle exponent notation? https://news.ycombinator.com/item?id=36027871
Earlier quoted context omitted.
I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the com…
It's best if your parser fails. Serde has an interface that allows failing. That one should fail. There is also another that panics, and AFAIK it will automatically panic on any parser that fails. Do not try to handle huge values, do not pretend your parser is total, and do not pretend it's a correct value. If you want to create an specialized parser that handles huge numbers, that's great. But any general one must f…
1e10 + 1e-10 = 10000000000.0000000001
1e10000000000000000000 + 1e-10000000000000000000 = ...
[EDIT]: Oh I see, fix the parser to disallow such large numbers from entering the system in the first place, then you don't have to worry about adding them together. Yeah that could be a good first step towards safety. Though, I don't know how to parametrize the serde call.
Earlier quoted context omitted.
It's best if your parser fails. Serde has an interface that allows failing. That one should fail. There is also another that panics, and AFAIK it will automatically panic on any parser that fails. Do not try to handle huge values, do not pretend your parser is total, and do not pretend it's a correct value. If you want to create an specialized parser that handles huge numbers, that's great. But any general one must f…
This isn't about parsing so much as letting the users do "dangerous" math operations. The obvious one is diving by zero, but when the library offers arbitrary precision, addition becomes dangerous with regard to allocating all the digits between a small and large value 1e10 + 1e-10 = 10000000000.0000000001 1e10000000000000000000 + 1e-10000000000000000000 = ... It's tough to know where to draw the lines between "safet…
Earlier quoted context omitted.
I was attempting to solve this very problem in the Rust BigDecimal crate this weekend. Is it better to just let it crash with an out of memory error, or have a compile-time constant limit (I was thinking ~8 billion digits) and panic if any operation would exceed that limit with a more specific error-message (does that mean it's no longer arbitrary-precision?). Or keep some kind of overflow-state/nan, but then the com…
I'd strongly recommend against this default - it's a major blocker for using the Haskell library with web APIs as it transforms JSON RPC into into readily available denial of service attacks. 8 billion digits (~100 bits?) is far more than should be used. Would it possible to use const generics to expose a `BigDecimal ` or `BigDecimal ` type with bounded precision for serde, and disallow this unsafe `BigDecimal` entir…
Having user-set generic limits would be cool, and something I considered when const generics came out, but there's a lot more work to do on the basics, and I'm worried about making the interface too complicated. (And I don't want to reimplement everything.) D
I also would like a customizable parser struct, with things like localization, allowing grouping-delimiters and such (1_000_000 or 1'000'000 or 10,00,000). That could also return some kind of OutOfRange parsing error to disallow "suspicious" values, out of range. I'm not sure how that to make that generic with the serde parser, but I may some safe limits to the auto serialization code.
Especially with JSON, I'd expect there's only two kinds of numbers: normal "human" numbers, and exploit attempts.
Earlier quoted context omitted.
This isn't about parsing so much as letting the users do "dangerous" math operations. The obvious one is diving by zero, but when the library offers arbitrary precision, addition becomes dangerous with regard to allocating all the digits between a small and large value 1e10 + 1e-10 = 10000000000.0000000001 1e10000000000000000000 + 1e-10000000000000000000 = ... It's tough to know where to draw the lines between "safet…
If you are using a library with this kind of number representation, computing any rational number with a repeating decimal representation will use up all your memory. 1/3=0.33333… It will keep allocating memory to store infinite copies of the digit 3. (In practice it stores it using binary representation but you get the idea.)
But for addition, the idea is to give the complete number if you do `a + b`, otherwise you could use the context to keep the numbers within your `ctx.add(a, b)`. But after the discussions here, maybe this is too unsafe... and it should use the default precision (or a slightly larger one) in the name of safety? With a compile time flag to disable it? hmm...