Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

App Engine down

code.google.com

Post reply on HN
Print view View original

11–20 of 139 posts

Re: App Engine down

#11

Post by Achshar »

josh2600 wrote: post #10
libria wrote: post #7

Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?

Build redundancy into your software to deal with single provider failure.

That is not always a feasible option, specially for young projects with limited capital.

Re: App Engine down

#12

Post by debacle »

I'm really happy I don't host in the cloud. How quickly are the cost savings of cloud computing obliterated by PR, customer service, and system administration time when an outage like this occurs?

Re: App Engine down

#13

Post by bsaul »

I would love it so much to see people at google showing all the internal tools they're using to detect and solve this kind of issues. I can only imagine a war room with screens all over the place showing gigantic amount of red flashing lines :) Hope it doesn't last for long though, i was just praising what a good choice app engine has been so far 10 minutes ago...

Re: App Engine down

#14

Post by bad_user »

libria wrote: post #7

Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?

Downtimes happen regardless, unless you have a lot of money and talent to spend on your own infrastructure and even then it's hard to beat cloud providers like Amazon, or Google, who have more resources and knowledge than you do.

The greatest thing about cloud-hosting is that you can just sit by and let them fix it. It usually takes about half an hour, or a couple of hours if the outage is severe, but usually less than the time it takes for an update of DNS records (unless you've got some proxy in front of your IPs, which would be another point of failure).

And then, even with these severe outages, the overall monthly uptime is still better than %99.9 and it's really hard to beat that, so just relax and let them fix it.

Re: App Engine down

#18

Post by davidjgraph »

libria wrote: post #7

Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?

I'd say the answer depends on how fast GAE recovers. If you're building redundancy over multiple clouds, if there's a lot of data:

1) It's very complex and expensive 2) You're looking at DNS to hot failover, in most cases.

If GAE can recover in less than 30 minutes and sticks to, say, one outage a year, you just can't justify the kind of cost you're looking at with 2 (seriously, it's a lot of cash).

Re: App Engine down

#19

Post by tomnewton »

My Google contact said that 'SRE are all over it. Hope to have more details soon.' but that was about 30 minutes ago.

Does tumblr.com use app engine? They're down...

Re: App Engine down

#20

Post by untog »

davis_m wrote: post #17

I think this is larger than just GAE. http://internettrafficreport.com/namerica.htm It seems like large portions of the internet are down.

Yes, Tumblr is experiencing difficulties and AFAIK they don't run anything on App Engine.
Post reply on HN