Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit's photo albums broke due to Integer overflow of Signed Int32

old.reddit.com

Post reply on HN
Print view View original

61–70 of 142 posts

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#61

Post by HideousKojima »

rezonant wrote: post #58
_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

> Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time. Signed integers have 2^31 positive values (well, just about). It doesn't take 16 bits to store the negative sign :-)

Not for most implementations, a signed 32 bit int goes from ~-2 billion to ~+2 billion, an unsigned from 0 to ~4 billion

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#62

Post by ipython »

josephg wrote: post #35
WhitneyLand wrote: post #26

Earlier quoted context omitted.

“32-bit IPv4 addresses to 128-bit IPv6 addresses means going from 4.3 billion to 340,282,366,920,938,463,463,374,607,431,768,211,456” Yes 128 bits is only 4x times the bit length, but the address space is exponentially bigger. Some predict if we do run out, it might take ~100 years.

> Some predict if we do run out, it might take ~100 years. You think we can use 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses in 100 years? Thats ... ahem, ambitious!

The truth is that the address space is extremely sparse. Every ipv6 subnet is a /64 by default, yet I’m sure most of us have vastly fewer than 2^64 machines in our networks … at least for now

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#63

Post by josho »

kristopolous wrote: post #23
monsieurbanana wrote: post #13

Earlier quoted context omitted.

To people reading this: please never ever do that, unless it was agreed upon with your client/employer beforehand (and compensated accordingly). If you know the code may break in 5 years, why don't you fix it now? Because your employer doesn't want to pay you to do that (probably for good reason, surely there's higher priorities). Then why would you do it for free years later?

No. You can say, anticipate breaking changes that is calendared to happen as an eventually rollover or deprecation Essentially things have to work one way now and there's solid reason to think it will no longer work that way later and there's no valid path to fix because exogenous conditions change. This alternatively, can be done in bad faith to guarantee a future paycheck by intentionally placing timebombs in and t…

The employer also shouldn't take you up on the offer because now you are engaging in work without a contract, which opens up contract issues, probably breaks things like SOC compliance, etc.

I like your accountability, but it's sufficient to document the issue in code, open a jira (or equivalent) and move on.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#64

Post by nonethewiser »

kristopolous wrote: post #5
fdgsdfogijq wrote: post #4

Earlier quoted context omitted.

Someone probably joked they would never reach that scale when they wrote that code

Thinking "if that ever gets anywhere close to a problem we'll have vast resources and plenty of time to fix it" and then, I'm guessing, that person left a few months later and nobody owned that part of the code because it worked. Then 10 or so years went by... Whenever I write code like that which may break in say, 5 years, I'll sign it in the comments and put my personal email and phone number inviting future people…

Hm. I wonder why no one ever called you.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#65

Post by _gabe_ »

loteck wrote: post #54
_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

presumably Isn't the answer to your question buried in your assumptions? For example, you seem to assume the rate of comsumption of unique IDs is static.

It would have to increase A LOT to significantly change the outcome. For example, according to this website[0] (no idea how accurate it is) Facebook users upload more than 300 million photos a day. At that rate, 2^32/300,000,000 = 14.3. So 32 bits would give Facebook 14 days worth of unique ids for their photos. Whereas 2^64/300,000,000 = 61,489,147,000 days, which is around 168 million years worth of ids.

All I'm saying is the jump from 2^32 to 2^64 is astronomical. I don't see using 64 bit integers for uids in my hobby code as something to be concerned about. In production code for a company I would use something more significant, but even then I feel like 64 bits will last a very long time.

[0]: https://bernardmarr.com/how-much-data-do-we-create-every-day...

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#66

Post by _gabe_ »

jxf wrote: post #59
_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

> Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time. Small correction: Signed int32s means you have 31 bits for the integer value (2^31 values), not 16 bits (2^16 values). There are 32 bits; 1 bit is used up for the sign, and 31 remain for the number.

Yep, I made the same mistake I was trying to call out here haha. Thanks for the correction!

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#67

Post by vikingerik »

johnfarrelldev wrote: post #41
boosteri wrote: post #27

Nostalgic flashback to Premier Manager games, where players stats decreased as they aged. When they went /below 0/ they flipped around to 127. So a good strategy was to scout out really bad players from lower leagues about to hit age 30+. And offer them very long contracts to prevent them retiring .. and give time for most of their stats to flip around, turning them into superstars.

I always found the funniest occurrence of this was in the Civ game though it seems it originally being a bug is disputed. https://en.wikipedia.org/wiki/Nuclear_Gandhi

The bug never existed at all in Civ 1. It was an urban legend all along. Similar behavior was intentional in Civ 5 as a joke, which convinced everyone that it really did happen in Civ 1 when it never did.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#68

Post by aidenn0 »

chrismcb wrote: post #60
_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

I think it is a horrible assumption to think that Reddit won't grow and will take exactly the same length of time to double the number of unique ids required. But still 64 bits should be plenty for a while. And presumably if it comes closer to using that many they'll have enough time to switch to 128

2^63 (assuming signed 64 bit ints) is enough for 10 billion people to post 900 million posts each. Short of faster-than-light travel or the singularity happening, it seems pretty safe.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#69

Post by pwdisswordfish0 »

liquidgecka wrote: post #20
madrox wrote: post #3

A long time ago we discovered Twitter used a round robin of three servers for assigning IDs to tweets. We inferred the round robin was done by doing mod 3 of a signed int32, and because that space doesn't divide neatly by two it meant one of the three servers saw less load than the others and we could map ID assignment volume according to how often it overflowed and hence estimate total tweet volume for a given perio…

I am pretty sure that snowflake didn't use a mod of a signed int32. It used a service discovery pool as part of finagle (and prior to that dns iirc). The server used a very simple method internally to convert time into a integer (that was 52 bits because of javascript). In fact it was completely open source: https://blog.twitter.com/engineering/en_us/a/2010/announcing... The integer generation was pretty simple, ther…

> 52 bits because of javascript

But IEEE 754 doubles have a significand that supports a 53-bit range. What am I missing?

Post reply on HN