Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

App Engine down

code.google.com

Post reply on HN
Print view View original

81–90 of 139 posts

Re: App Engine down

#81

Post by stickfigure »

debacle wrote: post #57
tomgallard wrote: post #23

Earlier quoted context omitted.

Surely hosting yourself exposes you to just as much, if not more risk? Problem in the datacentre where you're co-lo'd, or one of your servers blows up? I think people not trusting the cloud is similar to how people feel safer driving their cars then taking a plane. The stats say the plane's safer, but people prefer being in control. People like the idea of being in control of their servers, even if that means there's…

We're coloed across three datacenters spanning the US (one might be in TO I think) and if a datacenter were to go down, we have a hot backup that's no more than 12 hours stale. The only real manual maintenance that we've got is a rolling reimaging of servers based on whatever's in version control, which usually takes a few hours twice a year, but we'd probably do that if we were in the cloud anyway. When you can scri…

When you can script away 90% of your system administration tasks, hosting in the cloud doesn't really make a ton of sense.

How big is your ops team? I'm guessing it's more than 0.

Re: App Engine down

#82

Post by mikegioia »

davidjgraph wrote: post #6

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.

Re: App Engine down

#83

Post by fidotron »

lurker14 wrote: post #65
fidotron wrote: post #55

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

Which is better? Having a day of downtime each year, or not launching at all?

It's not just the cloud itself, but your connection into it. I'm more concerned that my connection to the internet is lost so many times a day (traveling underground etc.) and that this is enough to prevent many devices in my vicinity from acting in a coherent way. They should be able to synchronize among themselves without the central intermediary.

Obviously this doesn't happen because it's hard, but also companies have a vested interest in piping all sorts of data through them for analytics purposes. This is not in the interest of the users at all.

Re: App Engine down

#84

Post by acdha »

davidjgraph wrote: post #6

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought.

The correct less to draw is that any one point of infrastructure is a risk, so you need to scale wide. This is possible to do with AWS regions, or other providers - even internal bare iron if you're so inclined, but impossible to do with GAE because you're committed to a single-vendor API as well as their infrastructure.

Re: App Engine down

#85

Post by acdha »

Alan01252 wrote: post #42
tomgallard wrote: post #23

Earlier quoted context omitted.

Surely hosting yourself exposes you to just as much, if not more risk? Problem in the datacentre where you're co-lo'd, or one of your servers blows up? I think people not trusting the cloud is similar to how people feel safer driving their cars then taking a plane. The stats say the plane's safer, but people prefer being in control. People like the idea of being in control of their servers, even if that means there's…

I agree with you. From my past experience any data center is subject to risks. I've witnessed: Power failures. Cross site links being cut due to engineering works. Over heating due to air conditioning failures. Flooding And I've experienced all the above from a very large, very well known, very expensive data center company based in London.

This is true of every data center I've worked with. Also network providers: everyone has downtime and sometimes you learn the hard way that despite being written into your contract someone took the cheap way and ran your “redundant” fiber through the same conduit which a backhoe just tore up.

Re: App Engine down

#86

Post by peterwwillis »

davidkatz wrote: post #80
peterwwillis wrote: post #71

Earlier quoted context omitted.

Which is better? Using a fallacious comparison to suggest cloud computing is the only viable option, or comparing the pros and cons of different computing models to choose the best one for you?

While the argument was perhaps coming on a bit too strong, it's hard to deny the ease of deployment on cloud services. It's probably a safe bet to say that for most early stage startups the cloud is a good move.

I'm at a loss in these discussions. I don't understand this developer-point-of-view.

Can you specifically give me examples of why using a cloud provider is better for a startup than, for example, using a couple desktops in your garage?

You can't say it's because of backups because the cloud doesn't provide a backup (unless you purchase an extra data backup solution with your cloud provider?). And correct me if i'm wrong, but you still have to set up your development environment on your local computer to write the code, install libraries to test with, etc.

What exactly are the steps involved in "deploying" that you couldn't do on your laptop, or a VPS?

Re: App Engine down

#87

Post by acdha »

libria wrote: post #7

Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?

There's no such thing as "cloud downtime" - it's still servers, data centers, networks, same as everything else.

You need to decide how much uptime you're willing to pay for, how much your service can degrade for how long, and methodically address each level of the hierarchy between you and your customers – and it might be the case that you decide that the ongoing costs of your engineering support for e.g. wide geographic separation just aren't sustainable at the level your customers are willing to pay, particularly if you have something like a CDN helping keep your site partially responsive during less than catastrophic failures.

Re: App Engine down

#89

Post by Yoms »

Latest update:

"At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. The symptoms that service users would experience include slow response and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes."

https://groups.google.com/forum/?fromgroups=#!topic/google-a...

Re: App Engine down

#90

Post by warrentr »

kjhughes wrote: post #39

What's the earliest sign of trouble you've had? Pingdom reports my GAE-hosted site has been down since 2012-10-26 10:37:38 EST, a bit over an hour now. UPDATE: My site is back. Delayed report from Pingdom says site came back online after 50 minutes. Performance is sketchy still. We're probably not in the clear yet. At least we can now get to the status dash: http://code.google.com/status/appengine

I think the first tweet i saw was 7:35 PST
Post reply on HN