Live data from Hacker News

Reddit crashed because of the growing subreddit blackout

theverge.com

41–50 of 148 posts

Re: Reddit crashed because of the growing subreddit blackout

#41
post #29

> A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” Calling bullshit on that one. If they had expected stability issues, and knew exactly which subreddits will go private and when, how is it possible they had so much "anticipated issues"?

They have an outage every two weeks maximum so I also call bullshit on that one, Reddit hasn't exactly been the most stable website before.

Re: Reddit crashed because of the growing subreddit blackout

#42

Earlier quoted context omitted.

That makes sense to me. It would be one variant of a larger theory of "a subreddit going private is an expensive process that is poorly optimized because it's normally so rare". Another variant would be that clicking the "go private" button might do something like start a huge cascade of queries/calls to mutate the status of individual posts and comments (and possibly hence also trigger the massive cache invalidation…

Would a better protest be rapidly switching subreddits from public/private?

Too easy to shut down, since there's a finite number of mod enabled accounts.

Re: Reddit crashed because of the growing subreddit blackout

#44

I have not read a reddit devblog or anything like that. My guess is they are doing some kind of caching of posts via elasticsearch/redis or something similar. It must depend on the privacy settings of a sub. So when 1000 subs go private, they potentially had to update hundreds of millions of comments. Anyone else have a theory?

Posted elsewhere, but yeah that's my theory too (having actually helped built it in the first place). My guess is the caches got blown out because the entire system is built on the idea that most people are looking at the most popular content.

When you close down all the most popular content, you have to dig deep into the long tail for fresh content.

Also, my guess is that the code for building homepages is not optimized for having a lot of skips due to private Reddits, since most people have probably never been subscribed to a private reddit, or if they have it wasn't for very long, or even if for very long, never more than one or two at at time.

Re: Reddit crashed because of the growing subreddit blackout

#45

My unpopular theory is that the Reddit blackout increased Reddit usage, having the opposite of the intended effect.

It would certainly change usage patterns. And there are enough popular subreddits who didn't go in on the outage, so there's content to be had.

The adage "there's no such thing as bad press" could definitely apply, given how broadly covered this is (I mean, hell, I saw it show up on NPR).

Re: Reddit crashed because of the growing subreddit blackout

#47

How long before they admins swoop in and change ownership of the blacked out subreddits?

Finding new moderators would be a incredible challenge. If someone wants to be a moderators badly, that is usually disqualifying.

You get an immense amount of power with very little accountability except to the community. Put that in the wrong hands and you can destroy a community incredibly quickly.

Or, put power in the hands of a dilettante, and you get overrun with spam and insults very fast.

Re: Reddit crashed because of the growing subreddit blackout

#48

I have not read a reddit devblog or anything like that. My guess is they are doing some kind of caching of posts via elasticsearch/redis or something similar. It must depend on the privacy settings of a sub. So when 1000 subs go private, they potentially had to update hundreds of millions of comments. Anyone else have a theory?

Cache invalidation makes sense. Could also be some kind of fanout query that assumes the top subreddits are open and wastes CPU cycles.

It could even be that lots of queries to generate pages just fetch private posts and comments. The assumption was that users will not be subscribed to lots of subreddits that they have access to so skipping these posts is typically negligible and not worth using a more complex query for. If the fraction of skipped items becomes significant these queries will be doing lots more work. Its pretty believable that some very common query getting 25-50% more expensive will be more server capacity then Reddit has available.

Another possible explanation is that operations on private subreddits are not as optimized as it is expected that they have a small number of subscribers and take up a small amount of total resources. These assumptions would be flipped.

Re: Reddit crashed because of the growing subreddit blackout

#49

How long before they admins swoop in and change ownership of the blacked out subreddits?

The main reason why reddit works as a platform is because its mods and subreddit regulars do a lot of unpaid work. Where is reddit going to find enough people to work for free on such short notice?

They don't have to be good mods. There's precedent for a handful of mods to moderate a huge swath of subreddits.

Re: Reddit crashed because of the growing subreddit blackout

#50

How long before they admins swoop in and change ownership of the blacked out subreddits?

Oh, that's easy, if they are looking for scabs to mod a million+ sub, I volunteer myself immediately. I've got a lot of great ideas that I'm really eager to try out.

The only thing that I'll be able to guarantee is that you'll really not like me as a mod.

Post reply on HN