Earlier quoted context omitted.
Can't prove it's not aliens.
Another sibling comment was added 2 minutes ago, where Reddit themselves state that the blackout is the cause: > According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge. Maybe you should think a bit harder abo…
Reddit.com appears to be having an outage
631–640 of 696 posts
Re: Reddit.com appears to be having an outage
#632Earlier quoted context omitted.
I never worked at this scale, but could it also be that different subs are horizontally scaled and with so many people reverting to the subs that are still open the load is unevenly balanced?
Good question! And few people get to work at this scale, so it's not an unreasonable guess. I'll join you in speculating wildly about this, since, hey, it's kind of fun. IMHO sharding traffic by subreddit doesn't pass the smell-test, though. Different subreddits have very different traffic patterns, so the system would likely end up with hotspots continuously popping up, and it'd probably be toilsome to constantly ba…
The founding Reddit team was non-technical (Even Spez. I've been to UVA; it's not an engineering school; and spez had never done any real engineering before coming onto Reddit). They ran a skeleton crew of some 10ish people for a long time (none from any exceptional backgrounds. One of them was from Sun & Oracle, after their prime).
Same group that started with a node front-end, python back-end monolith, with a document-oriented database structure (i.e., they had two un-normalized tables in Postgres to hold everything). Later they switched to Cassandra and kept that monstrosity -- partly, because back then no one knew anything about databases except sysadmins.
Back then they were running a cache for listing retrieval. Every "reddit" (subreddit and the various pages, like front-page, hot, top, controversial, etc.) listing is kept in a memcache. Inside, you have your "top-level" listing information (title, link, comments, id, etc.). The asinine thing is that cache invalidation has always been a problem. They originally handled it using RabbitMQ queues: votes come in, they're processed, and then the cache is updated. Those things always got backed up, because no one thought about batching updates on a timer (or how to use lock-free) (and no one knew how to do vertical scaling, and when they tried, it made things even harder to reason about). You know what genius plan they had next to solve this? Make more queues. Fix throughput issues by making more queues, instead of fixing the root cause of the back-pressure. Later, they did "shard"/partition things more cleanly (and tried lock-free) -- but they never did any real research into fixing the aforementioned problem (how to handle millions of simple "events" a day... which is laughable thinking back to it now).
That's just for listings. The comment trees are another big bad piece. Again, stored un-normalized -- but this time actually batched (not properly, but it is a step up). One great thing about un-normalized databases and trees, is that there are no constraints on vertices. So a common issue was that you could back-up your queue (again) for computing the comment trees (because they would never get processed properly) (and you could slow the entire site to a crawl because your message broker was wasting its time on erroneous messages).
Later, they had the bright idea to move from a data center to AWS -- break everything up into microservices. Autoscaling there has always been bungled.
There was no strong tech talent, and no strong engineering culture -- ever.
-------
My 2 cents: it's the listing caches. The architecture around it was never designed to handle checking so many "isHidden" subreddits (despite that they're still getting updates to their internal listings) -- and it's coming undone.
Re: Reddit.com appears to be having an outage
#633Earlier quoted context omitted.
Another sibling comment was added 2 minutes ago, where Reddit themselves state that the blackout is the cause: > According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge. Maybe you should think a bit harder abo…
If they anticipated it, why did it become an issue?
Re: Reddit.com appears to be having an outage
#634Earlier quoted context omitted.
And it would in turn be worth noting that the creators of reddit had a philosophical and political commitment to free speech that drove their light-touch approach to moderation. It's not like the existence of that subreddit is evidence of an endorsement on their part.
Hi! I've done a bunch of trust and safety work and I see this trope a fair bit. Please help me understand what the difference is between, say, platforming racist harassment because of a "political commitment to free speech" and platforming racist harassment because you just kinda like racist harassment? I get that it might be different in the heads of the people who have worked very hard to create those platforms. I'…
The difference is intent. Intent matters. Intent is the difference between murder and manslaughter, or between a conspiracy and mere speech.
Re: Reddit.com appears to be having an outage
#635Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry
HN getting hugged, too.
Re: Reddit.com appears to be having an outage
#636Earlier quoted context omitted.
Yeah, I suppose there is some ambiguity there. Though I'd argue a standard like "does this harm people?" would even more open to interpretation and prone to abuse. Just about any idea can be framed as "harmful" to some person or group given a sufficient level of motivated reasoning (in fact, almost all modern cases of mass censorship seem to try to justify themselves that way). I much prefer a clear principle with fe…
In your "ideas" phrasing the exceptions seem implicit rather than explicit by virtue of not covering everything. I think it's better to be explicit about "you can do whatever you want, except [..]". That some people will try to abuse this seems inescapable no matter what; we'll still be argueing the details 200 or 2,000 years from now because there is no way to capture any of this in clear neat rules. The best we can…
Re: Reddit.com appears to be having an outage
#637Earlier quoted context omitted.
"Hey check out this website I discovered - it's called Slashdot" "It's called what!?" "Slashdot" "How do you spell that?" " http://slashdot.org " "Hang on, what? How many slashes and dots?"
So the conversation of what might happen occurred before the site was created?
I probably got the telephone part mixed up but was right about being confusing to read aloud.
Can’t get a better source that /. itself!
Re: Reddit.com appears to be having an outage
#638Lots of speculation on Twitter about this—a failed attempt to re-open all closed subreddits and instate their own moderators. I can't imagine it'd be that, although I do enjoy the conspiracy, and more likely they were using the window of reduced traffic to make some larger changes and they went awry
Re: Reddit.com appears to be having an outage
#639Earlier quoted context omitted.
My personal guess is it's down on purpose so they can say only 5% of subs who said they would go dark actually did, we just happened to have a service outage that day so they can push their own narrative t investors. Spez doesn't care anymore, he's focused on that IPO payday and to tell with everyone else. He's a liability to the company now, but the board isn't acting.
Here's a site which tracks how many subs have been made private. It's actually crazy that it's up to 93% of them now: https://reddark.untone.uk/
Re: Reddit.com appears to be having an outage
#640Earlier quoted context omitted.
Pointless speculation. Reddit has constant outages of this exact nature almost weekly. There's nothing special about this one.
Reddit definitely doesn’t have outages like this almost weekly. Maybe 10+ years ago, but it’s rare now.