If anyone from reddit is reading this, I tried to sign up for Gold so I could log in, but I can't because signing up requires you to log in. I suspect I'm not the only person who would have happily bribed you to get to the head of the line. I think you guys left a lot of money on the table.
Reddit has now been down for 30+ hours
31–40 of 49 posts
Re: Reddit has now been down for 30+ hours
#32Earlier quoted context omitted.
Let me ask you. How does a company get into that state? Say you had 5 engineers, 1 billion page views a month and ongoing issues. Wouldn't you dedicate some resources to that? When one engineer leaves, wouldn't you worry a little? And then another, and another? Would you sit and do nothing? These events did not occur in a day. They occurred over a lengthy period of time. I can sympathize with the lone engineer becaus…
Conde Naste will not let them hire more people. Obviously we know where the fault lies then. We don't know what's going on so I think it's a bit unfair to speculate. But is there any evidence that Reddit is profitable? Would having a more stable platform honestly have much of an impact on that? Server issues aside, they seem to be doing pretty well in terms of page views and user acquisition.
Given reddit’s content and its demographic appeal, I’d guess it’s got about as much revenue potential as 4chan, which is to say, barely any at all. Perhaps even less, considering Condé Nast won’t let reddit run ads from the porn sites and male enhancement product vendors that keep 4chan afloat. There’s a reason reddit has to resort to Flash games and lolcats as placeholders for its ad space.
Re: Reddit has now been down for 30+ hours
#33Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
>If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story.
I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people.
Of course, I've never been involved with keeping a site the size of Reddit operational, and I suspect the number of people required to do the kind of migration you're suggesting is an obvious choice would require Reddit to at least quadruple their staff.
I guess they could do it now, but odds are they would end up having to put Reddit in read-only mode for a week to do a proper migration with as tiny a staff as they have. And it's not clear that just throwing dedicated hardware at the problem would solve it.
Re: Reddit has now been down for 30+ hours
#34Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
Apparently all of Reddit's problem is either Amazon or Conde Nast.
Re: Reddit has now been down for 30+ hours
#35Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
Reddit's current technical staff is composed of a single engineer and 2 ops (maybe 3, but I believe it's 2).
They simply don't have the manpower to do that.
Re: Reddit has now been down for 30+ hours
#36Earlier quoted context omitted.
If they have to let users back in slowly (which they do), why shouldn't it be ones that contributed financially to the site?
If it has to be based on a metric, it should be based on the Karma. "Who contributed the most for the site."
They had a bad outage around a year ago which was the first time I saw the "emergency read-only" notice. Back then, the metric was "age of account". Even though by that metric, I would have been one of the first users back, I really don't mind that the Gold users are being let in on a "biased random" basis. As ceejayoz said, "why shouldn't it be the ones that contributed financially"?
The three metrics mentioned so far all have issues, but I think I like yours the least.
Gold - "Thanks for helping fund the site. Hopefully your dollars will let us hire folks that will ensure this doesn't happen again." Age - "Thanks for sticking by our little site through all of our outages. We know you've had to put up with a lot, but we appreciate it." Karma - "Thanks for all the cats. We love cats."
I realize that I'm lumping a lot of users who post great informative content in with the "better drink my own piss" image macro crowd, but karma just doesn't seem like the best metric here.
It's also harder to come up with a good threshold for karma. Where do you draw the line? 10k karma? 100k? Are we only including comment karma or submission karma? Both?
At least the Gold members have, in general, unambiguously contributed to the site. I don't have a problem using that as the primary metric here.
(I should note that I'm a reddit plebeian in this regard. I have no golden dog in this fight.)
Re: Reddit has now been down for 30+ hours
#37Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
They moved from their own infrastructure to AWS because they thought it would be more reliable and because they don't have much manpower.
Re: Reddit has now been down for 30+ hours
#38I recently set up a website, and did consider AWS to host it. I decided not to, based on the problems Reddit had been having with AWS. I eventually got a VPS; looks like I made the right decision.
Counterpoint: We've been happily using AWS for several years now. We haven't had one second of downtime. We host across multiple availability zones, and have been pretty insulated. We are hosted in the eastern region (which was the one having issues yesterday), but managed to get through everything ok. We did lose some instances in one AZ, but failed back to our other instances without any incident. ANY provider is g…
If you do use AWS, however, it is vitally important to understand the tools they provide for redundancy, and their limitations. There are multiple availability zones for a reason, and although there was some multi-AZ downtime at the start, that recovered relatively quickly.
Re: Reddit has now been down for 30+ hours
#39I wonder what the users of Reddit are doing? going for a walk outside and enjoying the nature?
Re: Reddit has now been down for 30+ hours
#40Earlier quoted context omitted.
Reddit has 1 billion page views monthly let's cut them some slack.
Let me ask you. How does a company get into that state? Say you had 5 engineers, 1 billion page views a month and ongoing issues. Wouldn't you dedicate some resources to that? When one engineer leaves, wouldn't you worry a little? And then another, and another? Would you sit and do nothing? These events did not occur in a day. They occurred over a lengthy period of time. I can sympathize with the lone engineer becaus…
[deleted]