Live data from Hacker News

Reddit.com appears to be having an outage

news.ycombinator.com

631–640 of 696 posts

Re: Reddit.com appears to be having an outage

#631
post #464

Earlier quoted context omitted.

Can't prove it's not aliens.

Another sibling comment was added 2 minutes ago, where Reddit themselves state that the blackout is the cause: > According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge. Maybe you should think a bit harder abo…

If they anticipated it, why did it become an issue?

Re: Reddit.com appears to be having an outage

#632

Earlier quoted context omitted.

I never worked at this scale, but could it also be that different subs are horizontally scaled and with so many people reverting to the subs that are still open the load is unevenly balanced?

Good question! And few people get to work at this scale, so it's not an unreasonable guess. I'll join you in speculating wildly about this, since, hey, it's kind of fun. IMHO sharding traffic by subreddit doesn't pass the smell-test, though. Different subreddits have very different traffic patterns, so the system would likely end up with hotspots continuously popping up, and it'd probably be toilsome to constantly ba…

A good, but wrong, assumption is to assume Reddit's engineers know what they're doing.

The founding Reddit team was non-technical (Even Spez. I've been to UVA; it's not an engineering school; and spez had never done any real engineering before coming onto Reddit). They ran a skeleton crew of some 10ish people for a long time (none from any exceptional backgrounds. One of them was from Sun & Oracle, after their prime).

Same group that started with a node front-end, python back-end monolith, with a document-oriented database structure (i.e., they had two un-normalized tables in Postgres to hold everything). Later they switched to Cassandra and kept that monstrosity -- partly, because back then no one knew anything about databases except sysadmins.

Back then they were running a cache for listing retrieval. Every "reddit" (subreddit and the various pages, like front-page, hot, top, controversial, etc.) listing is kept in a memcache. Inside, you have your "top-level" listing information (title, link, comments, id, etc.). The asinine thing is that cache invalidation has always been a problem. They originally handled it using RabbitMQ queues: votes come in, they're processed, and then the cache is updated. Those things always got backed up, because no one thought about batching updates on a timer (or how to use lock-free) (and no one knew how to do vertical scaling, and when they tried, it made things even harder to reason about). You know what genius plan they had next to solve this? Make more queues. Fix throughput issues by making more queues, instead of fixing the root cause of the back-pressure. Later, they did "shard"/partition things more cleanly (and tried lock-free) -- but they never did any real research into fixing the aforementioned problem (how to handle millions of simple "events" a day... which is laughable thinking back to it now).

That's just for listings. The comment trees are another big bad piece. Again, stored un-normalized -- but this time actually batched (not properly, but it is a step up). One great thing about un-normalized databases and trees, is that there are no constraints on vertices. So a common issue was that you could back-up your queue (again) for computing the comment trees (because they would never get processed properly) (and you could slow the entire site to a crawl because your message broker was wasting its time on erroneous messages).

Later, they had the bright idea to move from a data center to AWS -- break everything up into microservices. Autoscaling there has always been bungled.

There was no strong tech talent, and no strong engineering culture -- ever.

-------

My 2 cents: it's the listing caches. The architecture around it was never designed to handle checking so many "isHidden" subreddits (despite that they're still getting updates to their internal listings) -- and it's coming undone.

Re: Reddit.com appears to be having an outage

#633
post #464

Earlier quoted context omitted.

Another sibling comment was added 2 minutes ago, where Reddit themselves state that the blackout is the cause: > According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge. Maybe you should think a bit harder abo…

If they anticipated it, why did it become an issue?

I do not work for Reddit. Maybe they have some press contact info on their page you could try? Or Twitter or something?

Re: Reddit.com appears to be having an outage

#634

Earlier quoted context omitted.

And it would in turn be worth noting that the creators of reddit had a philosophical and political commitment to free speech that drove their light-touch approach to moderation. It's not like the existence of that subreddit is evidence of an endorsement on their part.

Hi! I've done a bunch of trust and safety work and I see this trope a fair bit. Please help me understand what the difference is between, say, platforming racist harassment because of a "political commitment to free speech" and platforming racist harassment because you just kinda like racist harassment? I get that it might be different in the heads of the people who have worked very hard to create those platforms. I'…

> Please help me understand what the difference is between, say, platforming racist harassment because of a "political commitment to free speech" and platforming racist harassment because you just kinda like racist harassment?

The difference is intent. Intent matters. Intent is the difference between murder and manslaughter, or between a conspiracy and mere speech.

Re: Reddit.com appears to be having an outage

#635
post #6

Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry

HN getting hugged, too.

[deleted]

Re: Reddit.com appears to be having an outage

#636
post #603

Earlier quoted context omitted.

Yeah, I suppose there is some ambiguity there. Though I'd argue a standard like "does this harm people?" would even more open to interpretation and prone to abuse. Just about any idea can be framed as "harmful" to some person or group given a sufficient level of motivated reasoning (in fact, almost all modern cases of mass censorship seem to try to justify themselves that way). I much prefer a clear principle with fe…

In your "ideas" phrasing the exceptions seem implicit rather than explicit by virtue of not covering everything. I think it's better to be explicit about "you can do whatever you want, except [..]". That some people will try to abuse this seems inescapable no matter what; we'll still be argueing the details 200 or 2,000 years from now because there is no way to capture any of this in clear neat rules. The best we can…

Fair point. I agree with the sentiment of "you can do whatever you want, except [..]", in the sense that I think we should err on the side of personal freedom. To be clear, I don't think focusing on "the free exchange of ideas" means other freedoms aren't important, and I'm not proposing a constitutional amendment or anything. It's just that from a rhetorical perspective I prefer to use terminology that encourages the strongest possible interpretation of the argument I'm making, and I think, for me at least, "the free exchange of ideas" does that best for all the reasons I named in my original comment and its replies.

Re: Reddit.com appears to be having an outage

#637

Earlier quoted context omitted.

"Hey check out this website I discovered - it's called Slashdot" "It's called what!?" "Slashdot" "How do you spell that?" " http://slashdot.org " "Hang on, what? How many slashes and dots?"

So the conversation of what might happen occurred before the site was created?

See the FAQ for an official answer - https://slashdot.org/faq

I probably got the telephone part mixed up but was right about being confusing to read aloud.

Can’t get a better source that /. itself!

Re: Reddit.com appears to be having an outage

#638
post #6

Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry

That is an interesting point. Would assume your subreddit isn’t yours so at any point you could grow and generate content and users and Reddit could take it over and add in their own moderators?

Re: Reddit.com appears to be having an outage

#639
post #562
post #536

Earlier quoted context omitted.

My personal guess is it's down on purpose so they can say only 5% of subs who said they would go dark actually did, we just happened to have a service outage that day so they can push their own narrative t investors. Spez doesn't care anymore, he's focused on that IPO payday and to tell with everyone else. He's a liability to the company now, but the board isn't acting.

Here's a site which tracks how many subs have been made private. It's actually crazy that it's up to 93% of them now: https://reddark.untone.uk/

Subs that are supposed to go down. This don't list all sub. A own a small sub that is not there, for example.

Re: Reddit.com appears to be having an outage

#640
post #17

Earlier quoted context omitted.

Pointless speculation. Reddit has constant outages of this exact nature almost weekly. There's nothing special about this one.

Reddit definitely doesn’t have outages like this almost weekly. Maybe 10+ years ago, but it’s rare now.

I use Reddit daily and I get the outages pretty frequently (twice a month?)... From Brazil.
Post reply on HN