Live data from Hacker News

App Engine down

code.google.com

101–110 of 139 posts

Re: App Engine down

#101

Meanwhile... Gmail etc are working quite fine. So the claim that if you build on GAE you "take advantage of the same infrastructure used for Google services!!" starts to ring a bit hollow.

Or perhaps they are older than the affected infrastructure and just haven't been rewritten. Or perhaps they happen to not use the affected resource. In fact, affected services include developers.google.com and code.google.com, and to a lesser extent youtube.com.

Re: App Engine down

#102
post #57

Earlier quoted context omitted.

We're coloed across three datacenters spanning the US (one might be in TO I think) and if a datacenter were to go down, we have a hot backup that's no more than 12 hours stale. The only real manual maintenance that we've got is a rolling reimaging of servers based on whatever's in version control, which usually takes a few hours twice a year, but we'd probably do that if we were in the cloud anyway. When you can scri…

When you can script away 90% of your system administration tasks, hosting in the cloud doesn't really make a ton of sense. How big is your ops team? I'm guessing it's more than 0.

Ops team? We're a two man operation with occasional contractors.

Re: App Engine down

#103

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

Point well made. We operate our tonido relay server across the world using linode, softlayer, herztner and it currently supports couple of million devices and half a million users.In the last 2 years our downtime is less than 30 minutes. It is nil for end users since they migrate to the nearest relay server hosted offered by a different provider.

SPOF,security and control are the major issues with Iaas and pass offerings.

SPOF, security and control are the major problems for the Iaas and pass.

Re: App Engine down

#104

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

We shifted from services that fail often independently, to services failing rarely all at once. Clearly the former is going to be more noticeable and has greater societal impact, but as a business I'll take the latter any day.

Re: App Engine down

#105

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.

I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.

Re: App Engine down

#106
At about 7:30am US/Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. Many App Engine applications are experiencing slow responses and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. We are actively working on restoring service as quickly as possible.

We are posting regular updates to our downtime-notify list here: https://groups.google.com/forum/?fromgroups=#!topic/google-a...

Thanks, Christina, Google App Engine Product Manager

Re: App Engine down

#107
post #40

Earlier quoted context omitted.

Internet Traffic Report, while a nice concept, is unfortunately very misleading. Their sample size is extremely small, and most of those are permanently down. Have a look through their list of north american routers and find one of them where packet loss has gotten worse as their main overall graph for packet loss would suggest - I've just been through them all and couldn't find one.

Their baseline values are very misleading. But their relative metrics can still be useful. For example, it's very inaccurate to say that 51% of the internet is down. But it's precise to say that packet loss among the working nodes has increased about 30% in the last 24 hours, and sharply.

But that's likely 2 or 3 nodes, not a meaningful sample

Re: App Engine down

#108
post #84

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought. The correct less to draw is that any one point of infrastructure is a risk, so you need…

GAE applications are distributed across multiple data centers[1], so in theory you get "scale wide" automatically. Unfortunately it looks like there was some sort of flaw in the architecture. I believe this is the first systemwide failure of the HRD.

The real question is: Can you and your ops team build a "scale wide" system better than Google?[2] How much effort are you willing to put into it, when those development resources could be put into making features instead?

[1] For apps which use the high-replication datastore. Old (deprecated) master/slave apps are served out of a single datacenter.

[2] This is a reasonably serious question to ask. In GAE, you're one app among many, so a you-specific scaling solution might be easier and more robust than the generalized one Google builds. But not necessarily.

Re: App Engine down

#109

Earlier quoted context omitted.

I'm at a loss in these discussions. I don't understand this developer-point-of-view. Can you specifically give me examples of why using a cloud provider is better for a startup than, for example, using a couple desktops in your garage? You can't say it's because of backups because the cloud doesn't provide a backup (unless you purchase an extra data backup solution with your cloud provider?). And correct me if i'm wr…

For what it’s worth, Heroku’s PG Backups addon is free of charge. https://addons.heroku.com/pgbackups

We also perform our own backups regardless of PG Backups and use Continuous Protection to protect recent data, even in the event of irrecoverable volume failure: https://devcenter.heroku.com/articles/heroku-postgres-docume...

Re: App Engine down

#110
post #17

I think this is larger than just GAE. http://internettrafficreport.com/namerica.htm It seems like large portions of the internet are down.

May not be anything or may be..

I noticed a couple of days ago that some of our dns entries were mysteriously removed from level 3 servers which out of old habit are used for resolution (some of the ip's go back to uu-net/worldcom/mci)

Now the interesting bit is they were for private subnet ip's. They're working fine everywhere else.

Today the last of their dns servers removed the entries so I had users go to google (8.8.8.8) and all's well with our apps.

Level 3's entries for our external stuff is there, just the private subnet stuff is removed.

If others do this too and resolve with level 3....

edit: just found this: http://tracker.outages.org/reports/view/59

Post reply on HN