Amazon Web Services are down • Hacker News Board

Re: Amazon Web Services are down

#341

Post by rm445 » Fri, Apr 22, 2011, 6:11 AM UTC

pbhjpbhj wrote: post #334

jedberg wrote: post #249

Earlier quoted context omitted.

No, I said that we have spent all we can, and at this point we need development. However, in this case, the outage is not because of any issues with our setup, but with Amazon.

>" we have spent all we can " So is it a financial constraint with Amazon? Would you be suffering the same sorts of outages regardless of the technology on the backend or does AWS basically suck?

You have the wrong end of the stick, because you're missing the history of the story. Reddit have a weird budget when it comes to staffing costs versus operating costs due to their parent company's policies as a media comapny - so they have a decent budget but are massively understaffed.

Statements like the one you're quoting are in that context. Let's say you have an unlimited operating budget - you can come up with all kinds of wonderful plans for massive redundancy and zero downtime. But you can't make that happen if you're not allowed to hire any engineers or sysadmins! As far as I'm aware reddit are paying Amazon mucho dinero but still having irredeemable problems with the storage product, EBS. They are stuck on an unreliable service without the manpower to move off.

That's the story, as far as I can piece together from comments here and on reddit.

Re: Amazon Web Services are down

#342

Post by obfuscate » Fri, Apr 22, 2011, 7:31 AM UTC

Devilboy wrote: post #338

obfuscate wrote: post #324

Earlier quoted context omitted.

Not sure what you had in mind by "large", but instance storage goes up to 1.7TB: http://aws.amazon.com/ec2/instance-types/

The reason Reddit uses RAID10 is for performance, not disk size. A single instance storage device is just too slow for the Reddit database.

Many instance types have 2 or 4 virtual disks (presumably on different physical disks).

Re: Amazon Web Services are down

#343

Post by nso » Fri, Apr 22, 2011, 8:22 AM UTC

mikecane wrote: post #229

athesyn wrote: post #33

Earlier quoted context omitted.

My four-day weekend is already off to a bad start(UK here).

Is it a 4-day holiday in Norway too?! Opera Support Forums have been sketchy for hours and if they won't come back until Tuesday, I'm SOL with my Opera problems.

5-day

Re: Amazon Web Services are down

#344

Post by dangrossman » Fri, Apr 22, 2011, 10:57 AM UTC

greaterscope wrote: post #257

Wish were able to download our ebs snapshots, which are supposedly hosted on S3. What does everyone else do?

I take the snapshots. I also have servers send backups to each other each night. I also have a nightly cron job run and rotate backups of the most critical databases to an external drive on my home network. A Tonido Plug does that job (Ubuntu on a tiny ARM server in a plug that costs virtually nothing to run).

Now, some of the databases are simply too large or under too much load to take a live backup while the sites are running. Those I run on Amazon RDS with the MultiAZ feature enabled. There should be two copies of the database running at all times, both servers keeping a 3 day binlog for point-in-time backups, and making a nightly snapshot to S3. I have to rely on Amazon for that.

But I still take daily home backups of the most valuable individual tables off those servers, like user registrations and payment records. Even if I can't have off-site backups of the whole database, I'll have off-site copies of the part I'd need most in case of an Amazon-entirely-offline catastrophe.

Re: Amazon Web Services are down

#345

Post by pbhjpbhj » Fri, Apr 22, 2011, 1:53 PM UTC

rm445 wrote: post #341

pbhjpbhj wrote: post #334

Earlier quoted context omitted.

>" we have spent all we can " So is it a financial constraint with Amazon? Would you be suffering the same sorts of outages regardless of the technology on the backend or does AWS basically suck?

You have the wrong end of the stick, because you're missing the history of the story. Reddit have a weird budget when it comes to staffing costs versus operating costs due to their parent company's policies as a media comapny - so they have a decent budget but are massively understaffed. Statements like the one you're quoting are in that context. Let's say you have an unlimited operating budget - you can come up with…

Ah, you see from what I read on reddit I understood that the staff shortage was simply part of Conde Nast's unwillingness to spend money on reddit and that constant downtime issues were another facet of that same problem.

It's not making money and those looking after reddit don't want to ruin it with a huge money grab - instead taking a soft approach, first just begging for money, then adding in a subscription model (freemium anyone?) and more subtle advertising by way of sponsored reddits (/r/yourCompany'sProduct type stuff).

I understand they've been hit with more staff problems just recently despite having a new [systems?] engineer start with them.

So in your view EBS is the problem regardless of finance? That was the nut I was attempting to crack. TBH I didn't expect someone at reddit to stick their neck out and say "yeah Amazon sucks" but they might have confirmed that the converse was true and they were simply lacking the necessary finance to support the massive userbase they have.

Re: Amazon Web Services are down

#346

Post by olahav » Sat, Apr 23, 2011, 9:44 AM UTC

timf wrote: post #222

Some quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984

Yes - It is all business decisions. As someone said already an instance on AWS can cost up to 7X a machine you own on co-location. here is how outbrain manages it's multi datacenter architecture while saving on Disaster recovery headroom. http://techblog.outbrain.com/2011/04/lego-bricks-our-data-ce...