Earlier quoted context omitted.
Conde Naste will not let them hire more people. Obviously we know where the fault lies then. We don't know what's going on so I think it's a bit unfair to speculate. But is there any evidence that Reddit is profitable? Would having a more stable platform honestly have much of an impact on that? Server issues aside, they seem to be doing pretty well in terms of page views and user acquisition.
Given reddit’s content and its demographic appeal, I’d guess it’s got about as much revenue potential as 4chan, which is to say, barely any at all. Perhaps even less, considering Condé Nast won’t let reddit run ads from the porn sites and male enhancement product vendors that keep 4chan afloat. There’s a reason reddit has to resort to Flash games and lolcats as placeholders for its ad space.
Reddit has now been down for 30+ hours
41–49 of 49 posts
Re: Reddit has now been down for 30+ hours
#42Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
I agree with your overall point that having a business continuity plan is required. They should be able to answer questions like how do we minimize impact when we lose one cassandra node or two, ebs gets corrupted, a whole avail-zone disappears, etc. I'm just saying that AWS provides the building blocks for you to do that.
Re: Reddit has now been down for 30+ hours
#43Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
>If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story. I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people. Of course, I've never been involved with keeping a site the size of Reddit operational, and I suspect the number of people required to do the kind of migration you're suggesting is an obvious choice wo…
You can definitely have a good plan, and be able to execute on that plan, with fewer people. I was lucky enough to work in a team that was running the search backend for a big internet company. Tens of thousands of machines, constant code, config and data pushes, configuration management, os images, kernel, etc. The team was responsible for developing and running the tools needed to keep the site up even when a whole datacenter lost power and you had to shift load from one coast to the other. The size of that team was much lower than 10 people.
Obviously Reddit will have to invest in both infrastructure and human resources to get there, but it can be done.
Re: Reddit has now been down for 30+ hours
#44Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridic…
>If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story. I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people. Of course, I've never been involved with keeping a site the size of Reddit operational, and I suspect the number of people required to do the kind of migration you're suggesting is an obvious choice wo…
Does the size of their site matter? Perhaps. Does it require 10 engineers to get a DR strategy going and be able to implement it? No. If you clearly plan this ahead of time, if all the pieces are in place then no it will not take that many. If I was to take into account multiple pieces of infrastructure it could still be done. DR is not about getting the exact same infrastructure up. Its about keeping the business running, its about staying alive and in those situations usually you try the minimal that will get you there. Do I really need my back end reporting and emailing systems when shit hits the fan? No. Messages will queue up and when those systems come online later they'll take care of it.
Am I simplifying it? Perhaps so lets take an example.
In your current infrastructure you've got 100 servers, you got a couple load balancers, 40 webservers, 30-40 app servers and 20 database servers. Load balancers a dummy HA thing where something like haproxy, nginx, keepalived, lvs or something else runs, the second one is just the backup that takes over when the first dies. Those are static configs that dont take much to get up and running somewhere else. 40 webservers. So in my place of work each webserver is identical and any one of those can go down without affecting the other. Usually thats varnish, nginx, apache or something of the like in the front end maybe proxying off to rails apps, php or python. Each server has no dependency except on the database. Got some mid caching layer like memcached? Ok well you'd have one per webserver in a cluster and those things drop in and out with ease. App servers follow suit if you know how to build systems that are not tightly coupled and deal with asynchronous requests. Alot of it comes down to queues and message passing. Obviously there are some inter dependencies which you cant help and if you can solve some of it with intermediary haproxy balancing then great. So far I feel like all of these things can easily be setup on servers anywhere else and code can be frequently updated.
The real complexity is in data stores. I assume in companies with more than 20 machines they may have chosen multiple datastores for different purposes so it doesnt all live in one place. Yes that adds to the difficulty of things but you have to break things down and work on each one at a time. If you're smart you'll have used something that can be replicated, has multi master or is clustered in most cases you dont even have to worry about it because almost every popular datastore these days have those capabilities. Lets look at mysql. Most of the time people go with basic master slave replication and move to master with multiple slaves so you can offload reads. If you're replicating within the one place then theres no reason you cant replicate offsite to your warm standby. If you do maintain that offsite slave then you've got an upto the minute copy of your database in the event of catastrophic failure. Now thats one database down, what if your using redis, well that also has replication so I'm not seeing a problem there either. Mongodb, cassandra, oracle, it really doesnt matter what it is because you can maintain a copy.
At the end of the day its about planning, its about taking the time to already have 80% of the work done and getting the major pieces up and running. Theres no excuse. There really isnt. You think facebook's in one datacenter? Yea at one point they were but back then you really think they didnt have a backup plan? Ok they are big with lots of people but honestly I dont care if you've got 10 different datastores (dont do that) and each is half a terabyte on disk. I dont care what the complexity is, if you think its too complex to have a running backup thats your fault. Imagine google said, "eh yea the application is a bit complicated and we really cant have a backup somewhere else". What?!
Migrations, if you have no plan and are doing it from scratch yes that takes time. It took me a month from start to finish and thats only because I spent the first two trying to figure out how to get chef to automate most of it. But that was a migration and I had the luxury of time. If it was a DR scenario I'd have at max 8-10 hours before the MD would be yelling at me. Put all your databases on servers somewhere else, get the code up and running and have a strategy for switching.
Re: Reddit has now been down for 30+ hours
#45Earlier quoted context omitted.
Given reddit’s content and its demographic appeal, I’d guess it’s got about as much revenue potential as 4chan, which is to say, barely any at all. Perhaps even less, considering Condé Nast won’t let reddit run ads from the porn sites and male enhancement product vendors that keep 4chan afloat. There’s a reason reddit has to resort to Flash games and lolcats as placeholders for its ad space.
If reddit became NSFW with ads you'd see a huge drop in traffic from people at work (that don't have access to Adblock on their computers) and a lot more people not disabling it for reddit.com at home.
Re: Reddit has now been down for 30+ hours
#46I wonder what the users of Reddit are doing? going for a walk outside and enjoying the nature?
Re: Reddit has now been down for 30+ hours
#47Re: Reddit has now been down for 30+ hours
#48I wonder what the users of Reddit are doing? going for a walk outside and enjoying the nature?
Re: Reddit has now been down for 30+ hours
#49Earlier quoted context omitted.
If it has to be based on a metric, it should be based on the Karma. "Who contributed the most for the site."
They had a bad outage around a year ago which was the first time I saw the "emergency read-only" notice. Back then, the metric was "age of account". Even though by that metric, I would have been one of the first users back, I really don't mind that the Gold users are being let in on a "biased random" basis. As ceejayoz said, "why shouldn't it be the ones that contributed financially"? The three metrics mentioned so f…