They are sort of limping back up now. Some users -- seems to be mostly gold users right now -- can get in.
It is reddit. Not experts-exchange. If it were "gold users can now login" that would be bad.
Reddit has now been down for 30+ hours
11–20 of 49 posts
Re: Reddit has now been down for 30+ hours
#12Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridiculous. If my company were to suffer outages on a weekly basis, my boss and I would both be under the gun and seriously before it even got to that point we would be restructuring systems and shifting hosting platforms. Before anyone says to me, "hold on, you cant just go moving infrastructure". BS. When your business is the website and you are providing a service, availability is damn well key. I've moved datacenters more than once, I've delt with disaster recovery, I know how to do these things transparently and before you start thinking I'm arrogant well actually I only have 4 years of experience in the system administration game so I'd expect people with more talent than me to be able to get the job done pretty damn effortlessly. If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story. That means switching dns and pushing to warm standby servers.
Engineers of reddit, take some initiative. Move off aws. I dont care how hard it is, look at the outages you've suffered. Look at the problems you've had. Havent you figured it out yet? Get some physical hardware with rackspace and sort it out.
Please understand this comment is not meant to attack anyone but I am frustrated with people shifting blame and not taking responsibility. If your down 30 hours then you have failed.
Re: Reddit has now been down for 30+ hours
#13Sadly, my productivity has not improved one bit :(
Re: Reddit has now been down for 30+ hours
#14I was wondering why I was logged in, able to comment and vote. They're letting back in members slowly, with Reddit Gold members having a larger chance of being allowed in: http://www.reddit.com/r/reddit.com/comments/gv63r/nobody_can... . Be sure to read the comments by jedberg, one of the reddit staff. Edit: Now the top of reddit says: >UPDATE: We are slowly getting our capacity back and as such allowing a random sub…
And aren't those Reddit Gold members pleased with themselves:
Link: http://www.reddit.com/r/reddit.com/comments/gv6pg/only_reddi...
Re: Reddit has now been down for 30+ hours
#15Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
Reddit has 1 billion page views monthly let's cut them some slack.
Re: Reddit has now been down for 30+ hours
#16Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
Reddit have real problems with staffing. They have said they are planning to do stuff like this, but their parent company Conde Nast won't let them hire anyone, for some corporate reason. They only have like one sysadmin IIRC.
Re: Reddit has now been down for 30+ hours
#17Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
Reddit has 1 billion page views monthly let's cut them some slack.
Let me ask you. How does a company get into that state? Say you had 5 engineers, 1 billion page views a month and ongoing issues. Wouldn't you dedicate some resources to that? When one engineer leaves, wouldn't you worry a little? And then another, and another? Would you sit and do nothing? These events did not occur in a day. They occurred over a lengthy period of time. I can sympathize with the lone engineer because I've been there. I still sleep with my laptop and phone next to my head. I know what the burn out feels like, so yes I feel for that engineer. But as a business letting it get to that point.
edit: JonnieCache mentioned Conde Naste will not let them hire more people. Obviously we know where the fault lies then.
Re: Reddit has now been down for 30+ hours
#18Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
[deleted]
Re: Reddit has now been down for 30+ hours
#19Earlier quoted context omitted.
Reddit has 1 billion page views monthly let's cut them some slack.
Let me ask you. How does a company get into that state? Say you had 5 engineers, 1 billion page views a month and ongoing issues. Wouldn't you dedicate some resources to that? When one engineer leaves, wouldn't you worry a little? And then another, and another? Would you sit and do nothing? These events did not occur in a day. They occurred over a lengthy period of time. I can sympathize with the lone engineer becaus…
> JonnieCache mentioned Conde Naste will not let them hire more people.
Yeah, I imagine the situation is more complicated than that, but look into some of the reddit blogposts and some of the admins comments on here for more details. There was a lot of stuff in the big 'AWS is down' thread about it, something about CN giving them an unlimited operating budget but a tiny, non negotiable hiring budget. God knows.
Re: Reddit has now been down for 30+ hours
#20I'm kindof curious how they're going to deal with the self-serve ads that were supposed to run yesterday. Just a straight refund, or...? /me patiently waiting for it to come back up so that I can buy some for next week... :(
jedberg said yesterday he will add a day to the self-serve ads.