Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit's photo albums broke due to Integer overflow of Signed Int32

old.reddit.com

Post reply on HN
Print view View original

51–60 of 142 posts

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#51

Post by _gabe_ »

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which seems like a safe bet.

So am I being naive when I use 64 bit values for unique IDs? Or is it actually plausible that 64 bits is plenty of information for centuries to come?

Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#53

Post by not2b »

monsieurbanana wrote: post #13
kristopolous wrote: post #5

Earlier quoted context omitted.

Thinking "if that ever gets anywhere close to a problem we'll have vast resources and plenty of time to fix it" and then, I'm guessing, that person left a few months later and nobody owned that part of the code because it worked. Then 10 or so years went by... Whenever I write code like that which may break in say, 5 years, I'll sign it in the comments and put my personal email and phone number inviting future people…

To people reading this: please never ever do that, unless it was agreed upon with your client/employer beforehand (and compensated accordingly). If you know the code may break in 5 years, why don't you fix it now? Because your employer doesn't want to pay you to do that (probably for good reason, surely there's higher priorities). Then why would you do it for free years later?

Sometimes it's because they don't know it will break in 5 years. They weren't anticipating exponential growth and when they started, the i32 range was a thousand times more than they needed. Ten more doublings later ...

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#54

Post by loteck »

_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

presumably

Isn't the answer to your question buried in your assumptions? For example, you seem to assume the rate of comsumption of unique IDs is static.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#55

Post by ryanianian »

hu3 wrote: post #15
btown wrote: post #11

Earlier quoted context omitted.

Fun fact: if you do a lot of INSERT... ON CONFLICT calls in Postgres from automated systems that are updating much more often than you insert, your autoincrement primary key can increment far far faster than your data volume (since it doesn't de-increment on a conflict) and overflow an int, grinding things to a halt. One of the more maddening outages I've had to deal with!

Similar for MySQL. If you open a transaction, INSERT with AUTO_INCREMENT, then rollback the transaction, no data is saved, except the auto generated id is used and the next INSERT uses id+1.

Same with sequences in Oracle. They don't participate in transactions. Looking at the gaps between "consecutive" IDs shows surprisingly interesting behavior.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#56

Post by donut »

madrox wrote: post #42
liquidgecka wrote: post #36

Earlier quoted context omitted.

I think what your describing was actually a problem that I introduced in our bind configuration back when we used DNS for service discovery. Its not exactly what your describing but I can totally see how it would appear that way externally. So initially we used 2 bind servers with authoritative zones serving using round robin. This worked fairly well as the load on each server was high enough to keep the round robin…

This is awesome and completely retconning a 10 year old project for me. I was working on social media analytics at Disney and we were exploring ways to measure twitter conversation of our brands, which led to us attempting to estimate total conversation volume, which is why this technical nuance was relevant to us. It was a wildly experimental time. Thanks for the story!

This thread is amazing.. thank you both for sharing!

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#57

Post by lilyball »

_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

If 32 bits lasted 10 years, 33 bits would not last 20, as engagement is not a constant. If we look at the last 10 years, presumably there were a lot more posts in the second 5 years than in the first 5 years. I don't know what Reddit's actual user/engagement metrics show though. Certainly if they got to a stable point where they had approximately the same activity going forward as they have going back then it would, but that's not usually how social sites work.

That said, it would still take quite a while to get to 64 bits.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#58

Post by rezonant »

_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

> Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time.

Signed integers have 2^31 positive values (well, just about). It doesn't take 16 bits to store the negative sign :-)

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#59

Post by jxf »

_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

> Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time.

Small correction: Signed int32s means you have 31 bits for the integer value (2^31 values), not 16 bits (2^16 values). There are 32 bits; 1 bit is used up for the sign, and 31 remain for the number.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#60

Post by chrismcb »

_gabe_ wrote: post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

I think it is a horrible assumption to think that Reddit won't grow and will take exactly the same length of time to double the number of unique ids required. But still 64 bits should be plenty for a while. And presumably if it comes closer to using that many they'll have enough time to switch to 128
Post reply on HN