Live data from Hacker News

Reddit's photo albums broke due to Integer overflow of Signed Int32

old.reddit.com

111–120 of 142 posts

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#111
post #102

Earlier quoted context omitted.

Mandela effect

It's the result of people repeating what they read on the Internet, not what they personally experienced. This happens way too often

> This happens way too often

You're not wrong; but fyi it only happens in your originating universe. Over here we all experienced it.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#112
post #51

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which…

I think it is a horrible assumption to think that Reddit won't grow and will take exactly the same length of time to double the number of unique ids required. But still 64 bits should be plenty for a while. And presumably if it comes closer to using that many they'll have enough time to switch to 128

> have enough time to switch to 128

It appears that the original developer thought that for 32 bits :)

I say that as someone who inherited a system that allowed for 2^32 session references stored in the db. Lets just say that sessions were created a lot faster than anticipated for the amount of traffic we had.

So one fine Sunday morning in a November a long time ago we ran out.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#113

Earlier quoted context omitted.

That doesn't change anything... it's still round robin. You just stopped at an arbitrary number of 8 integers instead of 9.

It's not arbitrary, GP stated it was a 3 bit counter. In GGP (or something, not sure how far down in the thread we are), they were referring to a 32 bit int counter until overflow. If you bin each number from 0 to 2^32 - 1 by mod 3, you don't get 3 bins of equal sizes, 1 bin always comes out smaller.

Smaller by a single number at most, it's effectively insignificant. Not sure where the 3-bit counter assumption came from as the original post said 32bit signed int.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#114

Earlier quoted context omitted.

Not when the counter overflows back to 0. If it's a 3 bit counter, 0 is A again, not C.

The comment says a 32-bit signed int. Where is the 3-bit assumption coming from?

The comment starts with that assumption for the sake of a concise example. do you expect them to write the whole sequence out using a 32bit counter? :-D

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#115

Earlier quoted context omitted.

The comment says a 32-bit signed int. Where is the 3-bit assumption coming from?

The comment starts with that assumption for the sake of a concise example. do you expect them to write the whole sequence out using a 32bit counter? :-D

Then it's irrelevant. The point is that the remainders are evenly distributed as the integer is incremented, with at most a single round being short. That's just how the math works.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#116

Earlier quoted context omitted.

The comment says a 32-bit signed int. Where is the 3-bit assumption coming from?

The comment starts with that assumption for the sake of a concise example. do you expect them to write the whole sequence out using a 32bit counter? :-D

When you divide 2^32 across 3 machines, you get 1431655766, 1431655765, 1431655765. They're nearly identical, just an itsy bitsy bit different. This isn't going to cause one machine to have noticeably less load than the others.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#117
post #3

A long time ago we discovered Twitter used a round robin of three servers for assigning IDs to tweets. We inferred the round robin was done by doing mod 3 of a signed int32, and because that space doesn't divide neatly by two it meant one of the three servers saw less load than the others and we could map ID assignment volume according to how often it overflowed and hence estimate total tweet volume for a given perio…

I am pretty sure that snowflake didn't use a mod of a signed int32. It used a service discovery pool as part of finagle (and prior to that dns iirc). The server used a very simple method internally to convert time into a integer (that was 52 bits because of javascript). In fact it was completely open source: https://blog.twitter.com/engineering/en_us/a/2010/announcing... The integer generation was pretty simple, ther…

Someone previously created a tweet linking to itself by predicting the likely ID range: https://oisinmoran.com/quinetweet

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#118

Earlier quoted context omitted.

The comment starts with that assumption for the sake of a concise example. do you expect them to write the whole sequence out using a 32bit counter? :-D

Then it's irrelevant. The point is that the remainders are evenly distributed as the integer is incremented, with at most a single round being short. That's just how the math works.

"A single round being short" is exactly what they're talking about.

Edit to add: whether that's a big enough effect for the use case they're talking about, I don't know. This sort of thing is definitely significant in cryptographic code, though.

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#119

If I'm understanding the shitty change log exactly, was the solution to add an extra bit?

I guess thats a joke. Adding a single extra bit would usually be more complex then going to 64bit or going to 32bit unsigned.

Well, I say that, but actually, "adding an extra bit" is basically what going from signed to unsigned would do. So maybe they just added an extra (32nd) bit?

Re: Reddit's photo albums broke due to Integer overflow of Signed Int32

#120
post #9

Earlier quoted context omitted.

Strange thing is that photo albums re relatively new. Imgur was the go-to host for Reddit, and then they made their own uploader a looong time later. The "albums" functionality only came out in July of 2020, according to a Google search. Seems this was less likely a "someone else will deal with it" problem, and more of a development / QA testing problem.

For some reason most stuff still defaults to i32 and a lot of people use them for new code. At this point I'd not be against linters warning against using 32 bit ints unless you have a good reason.

They're still a good choice for most perf critical code due to the reduced memory use.
Post reply on HN