Ask HN: How has the Amazon AWS outage affected you? • Hacker News Board

Re: Ask HN: How has the Amazon AWS outage affected you?

#31

Post by benologist » Thu, Apr 21, 2011, 11:10 PM UTC

My leaderboards, user-created levels and other miscellaneous bits of Playtomic run on AWS via MongoHQ and I've spent 1/2 the day answering emails explaining the situation. On the plus side MongoHQ are planning to introduce the ability to mirror databases onto our own servers which is going to be frigging awesome.

It at least highlighted another issue - because users are trying to access databases that don't exist they're taking ages to time out which is bogging down the servers' other thousands of requests/second.

Re: Ask HN: How has the Amazon AWS outage affected you?

#32

Post by ig1 » Thu, Apr 21, 2011, 11:15 PM UTC

Startup's site has been down all day, probably lost about ~$500-$1000 in sales, more concerned about the longer term loss of goodwill and the potential damage to google seo rankings though.

Re: Ask HN: How has the Amazon AWS outage affected you?

#33

Post by dpcan » Fri, Apr 22, 2011, 6:23 AM UTC

bdclimber14 wrote: post #26

dpcan wrote: post #15

Earlier quoted context omitted.

Might as well get some good out of it.... please describe what your service is.

Sadly, I almost replied "just check out the website." I made a site for graphic design students to find freelance gigs that are posted by local small businesses who need cheap, small amounts of design work done. Thankfully this hasn't been very big, and mostly just a side project recently, so not too many users are being affected.

That's a great idea, can't wait to check it out when it's back.

Re: Ask HN: How has the Amazon AWS outage affected you?

#34

Post by jk215 » Fri, Apr 22, 2011, 2:33 PM UTC

api wrote: post #3

Reddit is down, so I got more work done today.

I woke up this morning in disbelief that it was still down. Its friday all I want to do is put off work and browse reddit.

Re: Ask HN: How has the Amazon AWS outage affected you?

#35

Post by kathryna » Fri, Apr 22, 2011, 7:48 PM UTC

I run an app that helps farms manage their weekly or monthly CSA programs. One farm had their monthly distribution yesterday - thankfully they had already downloaded the data they needed to pack and label the orders, but they weren't able to tell customers their final total or send reminders, so they had higher-than-average missed pickups and are receiving money several days later than usual. Farmers are already distrustful of digital/automated solutions, so I'm sad that this has added to that distrust.

Re: Ask HN: How has the Amazon AWS outage affected you?

#36

Post by callmeed » Sat, Apr 23, 2011, 5:39 AM UTC

I've got an app in private beta on Heroku. Closing in on 2 days of downtime.

Not a good way to instill confidence in the 50 or so potential customers I let into the beta.

Plus my iOS app gets data from a Heroku app that is down.

Re: Ask HN: How has the Amazon AWS outage affected you?

#37

Post by st3fan » Sun, Apr 24, 2011, 12:15 AM UTC

nostromo wrote: post #17

Our database (RDS) is completely inaccessible -- so our website has been down since 2am or so, showing our maintenance page. It's backed up -- but we have no way to download the most recent backup and move to another server. We tried rebooting our db about 11 hours ago -- and it's been stuck in reboot mode ever since. We tried creating a copy from a recent backup -- but it's stuck in boot mode. We can't move the snap…

"Offsite Backups" - Where Offsite means, not on Amazon's infrastructure. Seriously.

Re: Ask HN: How has the Amazon AWS outage affected you?

#38

Post by fomojola » Mon, Apr 25, 2011, 6:11 PM UTC

Saw the same RDS behavior nostromo saw: was actually truly interesting. After 2 days of arguing with the RDS database I actually spun up a new local storage instance, logged in over the mysql console at a shell (which actually worked, surprisingly enough) and then found out that a single table (this is a Wordpress instance) was hung: I guess the RDS instance couldn't access the disk file for that table (it was MyISAM which I guess isn't in memory?) so any requests for that specific table hung. As it so happens that was the wp_options table, so there was no way to get past that.

So I sat there with mysqldump and exported each table separately, then spun up a new MySQL instance in another AZ and then imported each table piecemeal. I then re-created the wp_options table from another dump (from an early prototype on Linode, actually) and then manually fudged the values till it all worked. That eventually worked for me.